the nerd's backyard -- 伪技术杂谈

2022年8月26日星期五

Install GitLab using Docker Compose

1. Install Docker Compose

# 1. remove installed & install dependency
sudo apt remove docker-desktop
sudo rm -r $HOME/.docker/desktop
sudo rm /usr/local/bin/com.docker.cli
sudo apt purge docker-desktop
sudo apt install gnome-terminal

# 2. setup the repository
sudo apt-get update

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 3. Install Docker

sudo apt-get update
sudo apt-get install docker-compose-plugin

2. Install GitLab

## 1. Preparation
# Go to the targeted folder
cd /mnt/data/Applications/gitlab

vi /etc/hosts
# private domain to IP
127.0.0.1    gitlab.example.com

vi .env
# set GITLAB_HOME in .env
GITLAB_HOME=/mnt/data/Applications/gitlab

vi docker-compose.yml
# docker-compose.yml
version: '3.6'
services:
  web:
    image: 'gitlab/gitlab-ee:latest'
    restart: always
    hostname: 'gitlab.example.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.example.com:1080'
        gitlab_rails['gitlab_shell_ssh_port'] = 1022
        # Add any other gitlab.rb configuration here, each on its own line
    ports:
      - '1080:1080'
      - '443:443'
      - '1022:22'
    volumes:
      - '$GITLAB_HOME/:/mnt/data/Applications/gitlab' 
      - '$GITLAB_HOME/config:/mnt/data/Applications/gitlab/config'
      - '$GITLAB_HOME/logs:/mnt/data/Applications/gitlab/logs'
      - '$GITLAB_HOME/data:/mnt/data/Applications/gitlab/data'
    shm_size: '256m'

## 2. docker compose installation
sudo docker --env-file .env up -d

# 3. get password for root
sudo docker exec -it gitlab-web-1 grep 'Password:' /etc/gitlab/initial_root_password

## 4. Browsing GitLab
## http://gitlab.example.com:1080 
## or 
## 192.168.0.xxx:1080

# reconfiguration
sudo docker exec -it gitlab-web-1 editor /etc/gitlab/gitlab.rb
# reconfigure as needed
# ...
sudo docker stop gitlab-web-1
sudo docker restart gitlab-web-1

TBC...

TBC... #TODO SMTP,

2021年11月24日星期三

Mysql on Ubuntu 20.04

## installation

sudo apt install mysql-server

pip install mysql-server

## set up security

sudo mysql_secure_installation

## login

sudo mysql -u root

## create user

CREATE USER 'user'@'localhost' IDENTIFIED BY 'password';

## Create mysql db

## Change mysql==8.0.27 password in Ubuntu 20.04.1

sudo /etc/init.d/mysql stop

(maybe): sudo mkdir /var/run/mysqld && sudo chown mysql /var/run/mysqld

sudo mysqld_safe --user=mysql --skip-grant-tables --skip-networking &

mysql -u root mysql

flush privileges;

ALTER USER 'root'@'localhost' IDENTIFIED BY 'Password=1234';

sudo killall -u mysql

sudo systemctl restart mysql.service

sudo mysql -u root -p

################## Clone DB ################

Make a dump of your source database:
```
mysqldump -uroot -p my_project -r my_project.sql
```
Or, if you only want to dump the database's table structure (schema) without any contents:
```
mysqldump -uroot -p my_project -r my_project.sql --no-data
```
Open up a MySQL shell:
```
mysql -uroot -p
```

From the MySQL shell, create a new database and populate it with the dumped data:

CREATE DATABASE my_project_copy;
USE my_project_copy;
SOURCE my_project.sql;

Create a user and give it permissions to the new database:

CREATE USER new_user IDENTIFIED BY 'some_password';
GRANT ALL ON my_project_copy.* TO 'new_user'@'localhost' IDENTIFIED BY 'some_password';

2019年11月22日星期五

Install TA-Lib on different OS

Ubuntu: #!/usr/bin/env bash
	## sudo apt install build-essential wget -y
	## wget https://artiya4u.keybase.pub/TA-lib/ta-lib-0.4.0-src.tar.gz
	## tar -xvf ta-lib-0.4.0-src.tar.gz
	## cd ta-lib/
	## ./configure --prefix=/usr
	## make
	## sudo make install wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz \ && sudo tar -xzf ta-lib-0.4.0-src.tar.gz \ && sudo rm ta-lib-0.4.0-src.tar.gz \ && cd ta-lib/ \ && sudo ./configure --prefix=/usr \ && sudo make \ && sudo make install \ && cd ~ \ && sudo rm -rf ta-lib/ \ && pip install ta-lib

Mac Os:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null
brew install ta-lib


For all:
pip install ta-lib

2018年8月13日星期一

xgboost & lightgbm with gpu on windows

Install latest visual studio and R 3.5.1

XGB: 1. git clone --recursive https://github.com/dmlc/xgboost

2. cmake .. -G"Visual Studio 15 2017 Win64" -T v140 -DUSE_CUDA=ON -DR_LIB=ON

3. cmake --build . --target install --config Release

4. install.packages() with local file

LightGBM:
1. http://lightgbm.readthedocs.io/en/latest/GPU-Windows.html
(address-model=64)
i.e., (CUDA, Rtools, Boost)

2. git clone --recursive https://github.com/Microsoft/LightGBM

cd LightGBM/R-package

3. Set use_gpu to TRUE in R-package/src/install.libs.R to enable the build with GPU support.

4. Rscript build_package.R
R CMD INSTALL lightgbm_2.1.1.tar.gz --no-multiarch

5. Install.packages in R with local file

2017年5月1日星期一

Install GUI of Ubuntu on AWS

1. Connect ssh to ec2 instance.

2. install vncserver:
sudo apt install ubuntu-desktop
sudo apt install vnc4server
sudo apt install gnome-panel

3. Type 'vncserver' to setup pwd

4. Kill vncserver 'vncserver -kill :1'

5. Type following:

vim /home/ubuntu/.vnc/xstartup

and modify

unset SESSION_MANAGER
# exec /etc/.....
gnome-session -session=gnome-classic &
gnome-panel&

('ESC', ':wq', 'ENTER' to quit vim)

Type 'vncserver' to start.

Download TightVNC Viewer to connect.

In AWS website, to set "Inbound rule" to allow connections, with IP::port

Tensorflow with AWS: https://aws.amazon.com/cn/blogs/china/tensorflow-on-aws/

2016年9月26日星期一

Ubuntu install lists

A list of post installation of Ubuntu

sudo apt-get install freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa libglu1-mesa-dev

sudo apt-get update

sudo apt-get upgrade

Google Chrome

http://cn.soulmachine.me/2016-08-17-deep-learning-cuda-development-environment/

Driver:

sudo add-apt-repository ppa:graphics-drivers/ppa

sudo apt-get update

sudo apt-get install nvidia-375 (gforce) -377(quadro)

1. 'stop x server'

'Ctrl + Alt + F1' ===> terminal

sudo service lightdm stop

2. download and install driver

sudo sh NVIDIA.......

Cuda  #install

sudo sh cuda_8.0.27_linux.run --tmpdir=/tmp --override

sudo sh cuda_8.0.27.1_linux.run

~/.bashrc

export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

#test

cd ~/NVIDIA_CUDA-8.0_Samples/1_Utilities/deviceQuery

make

./deviceQuery

cd ~/NVIDIA_CUDA-8.0_Samples/5_Simulations/nbody/

make

./nbody -benchmark -numbodies=256000 -device=0

Output:


> Windowed mode

> Simulation data stored in video memory

> Single precision floating point simulation

> 1 Devices used for simulation

gpuDeviceInit() CUDA Device [0]: "GeForce GTX 1080

> Compute 6.1 CUDA device: [GeForce GTX 1080]

number of bodies = 256000

256000 bodies, total time for 10 iterations: 2364.286 ms

= 277.192 billion interactions per second

= 5543.830 single-precision GFLOP/s at 20 flops per interaction

cuDNN

'copy ../cuda/include/*.h  to  /usr/local/cuda-8.0/include/'

'copy ../cuda/lib64/*.* to /usr/local/cuda-8.0/lib64'

**'/usr/local/cuda-8.0' is the path where CUDA installed.

Tensorflow

sudo apt-get install libcurl3-dev

$ conda install virtualenv # suggested by the output of terminal not using pip install
$ conda create --name=tensorflow_env python=3.5
$ source activate tensorflow_env
$ pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl # the code you provided is for the python 3

(to delete env: conda env remove --name tensorflow)

Rebuild tensorflow with CUDA!!!

https://alliseesolutions.wordpress.com/2016/09/08/install-gpu-tensorflow-from-sources-w-ubuntu-16-04-and-cuda-8-0-rc/

Note:

1. Install protobuf 3.0!!MUST BE 3.0

pip install protobuf3

2. MUST downloads tensorflow from online, NOT get from the command line!!!

In build:

$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
$ bazel-bin/tensorflow/tools/pip_package/build_pip_package ~/Downloads/tensorflow_pkg
$ sudo pip3 install ~/Downloads/tensorflow_pkg/tensorflow-0.10.0-py3-none-any.whl

######==========================================================================

Caffe:

http://caffe.berkeleyvision.org/installation.html#prerequisites

In compiling opencv, BEFORE 'make':

In file 'modules/cudalegacy/src/graphcuts.cpp' change:

-#if !defined (HAVE_CUDA) || defined (CUDA_DISABLER)

+// GraphCut has been removed in NPP 8.0

+#if !defined (HAVE_CUDA) || defined (CUDA_DISABLER) || (CUDART_VERSION >= 8000)

prereqisites:

1. protobuf

sudo apt-get install autoconf automake libtool curl make g++ unzip

2. glog, etc..

sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
3. hdf5

sudo apt-get install hdf5-tools

cd /usr/lib/x86_64-linux-gnu

sudo ln -s libhdf5_serial.so.10.1.0 libhdf5.so

sudo ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_hl.so

Changes in Makefile.config

Uncomment 'OPENCV_VERSION := 3', 'USE_CUDNN := 1', 'ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe/', 'PYTHON_INCLUDE := $(ANACONDA_HOME)/include \

  # $(ANACONDA_HOME)/include/python3.5 \

  # $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include \', 'USE_PKG_CONFIG := 1'

(BEFORE 'make all' for caffe)

find . -type f -exec sed -i -e 's^"hdf5.h"^"hdf5/serial/hdf5.h"^g' -e 's^"hdf5_hl.h"^"hdf5/serial/hdf5_hl.h"^g' '{}' \;

mkdir build
cd build
cmake ..
make all      (Or cmake -DWITH_IPP=ON . && make -j $(nproc) && make install???)
make install
make runtest

C++

CodeBlocks:

sudo add-apt-repository ppa:damien-moore/codeblocks-stable
sudo apt-get update

Anaconda

Pycharm:

sudo add-apt-repository ppa:mystic-mirage/pycharm

sudo apt-get update

sudo apt-get install pycharm-comminity

Notepad++:


sudo add-apt-repository ppa:notepadqq-team/notepadqq

sudo apt-get update

sudo apt-get install notepadqq

2016年3月16日星期三

Install Pyspark in Windows and with PyCharm!!!

Build Spark environment for Windows command line.
Paths added in this session are for Win environment by default.

1. Install java JDK to ..\Java\jdk1.8.0_74

Add path ..\Java\jdk1.8.0_74;..\Java\jdk1.8.0_74\bin

2. Install scala to ..\scala

Add path ..\scala\bin

3. DONOT!!! Install Python 3

3. Install Anaconda3 and ADD PATH the most front!!

4. Download the latest prebuilt Spark to ..\Spark\spark-1.6.1

Add path: ..\Spark\spark-1.6.1;..\Spark\spark-1.6.1\bin

5. SPARK_HOME = ..\Spark\spark-1.6.1

Download hadoop "winutils.exe" etc. files.

HADOOP_HOME = ../hadoop/hadoop-common-2.2.0-bin-master

Now java, scala, and spark can all be run in command line in windows.

======================================================================
Next, go further for PyCharm IDE
Paths added in this session are for "current-project" settings in Run\EditConfigurations.

6. Install PyCharm

7. Add Environment Variables in PyCharm IDE
PYTHONPATH = ../Spark/spark-1.6.1/python;/Spark/spark-1.6.1/python/lib/py4j-0.9-src.zip;..\hadoop\hadoop-common-2.2.0-bin-master\bin

HADOOP_HOME = ../hadoop/hadoop-common-2.2.0-bin-master

SPARK_HOME = Spark/spark-1.6.1

=======================================================================
To be further tested and simplified!

订阅：博文 (Atom)