Mount smb network shared folder

sudo apt update

sudo apt install cifs-utils

sudo mkdir /mnt/local_folder

sudo vi ~/.smbcredentials

username=xxx

password=xxx

sudo vi /etc/fstab

# add to last line:

//192.168.xx.xx/shared_folder /mnt/local_foler cifs credentials=/home/your_local_username/.smbcredentials,uid=1000,gid=1000,_netdev,vers=3.0 0 0

(uid = id -u your_local_username)

(gid = id -g your_local_username)

sudo systemctl daemon-reload

# test mount

sudo mount -a

Ubuntu Tricks

Copy with speed and progress:

 rsync -r -v --info=progress2 ssh user@remote-system:/address/to/remote/file /home/user/

Clear cache / buffer:

sudo sync && sudo sysctl vm.drop_caches = 1|2|3

or

echo 3 | sudo tee /proc/sys/vm/drop_caches

Periodically clear cache / buffer:

sudo crontab -e
0 * * * * /usr/bin/sync; /usr/sbin/sysctl vm.drop_caches=3

vfs cache threshold:

sudo sysctl vm.vfs_cache_pressure=100

Docker the Gitlab Server

Install GitLab using Docker Compose

1. Install Docker Compose

# 1. remove installed & install dependency
sudo apt remove docker-desktop
sudo rm -r $HOME/.docker/desktop
sudo rm /usr/local/bin/com.docker.cli
sudo apt purge docker-desktop
sudo apt install gnome-terminal

# 2. setup the repository
sudo apt-get update

sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
    "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 3. Install Docker

sudo apt-get update
sudo apt-get install docker-compose-plugin

2. Install GitLab

## 1. Preparation
# Go to the targeted folder
cd /mnt/data/Applications/gitlab

vi /etc/hosts
# private domain to IP
127.0.0.1    gitlab.example.com

vi .env
# set GITLAB_HOME in .env
GITLAB_HOME=/mnt/data/Applications/gitlab

vi docker-compose.yml
# docker-compose.yml
version: '3.6'
services:
  web:
    image: 'gitlab/gitlab-ee:latest'
    restart: always
    hostname: 'gitlab.example.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://gitlab.example.com:1080'
        gitlab_rails['gitlab_shell_ssh_port'] = 1022
        # Add any other gitlab.rb configuration here, each on its own line
    ports:
      - '1080:1080'
      - '443:443'
      - '1022:22'
    volumes:
      - '$GITLAB_HOME/:/mnt/data/Applications/gitlab' 
      - '$GITLAB_HOME/config:/mnt/data/Applications/gitlab/config'
      - '$GITLAB_HOME/logs:/mnt/data/Applications/gitlab/logs'
      - '$GITLAB_HOME/data:/mnt/data/Applications/gitlab/data'
    shm_size: '256m'

## 2. docker compose installation
sudo docker --env-file .env up -d

# 3. get password for root
sudo docker exec -it gitlab-web-1 grep 'Password:' /etc/gitlab/initial_root_password

## 4. Browsing GitLab
## http://gitlab.example.com:1080 
## or 
## 192.168.0.xxx:1080

# reconfiguration
sudo docker exec -it gitlab-web-1 editor /etc/gitlab/gitlab.rb
# reconfigure as needed
# ...
sudo docker stop gitlab-web-1
sudo docker restart gitlab-web-1

TBC...

TBC...  #TODO SMTP, 

Mysql on Ubuntu 20.04

## installation

sudo apt install mysql-server

pip install mysql-server

## set up security

sudo mysql_secure_installation

## login

sudo mysql -u root

## create user

• CREATE USER 'user'@'localhost' IDENTIFIED BY 'password';

## Create mysql db

## Change mysql==8.0.27 password in Ubuntu 20.04.1

sudo /etc/init.d/mysql stop

(maybe): sudo mkdir /var/run/mysqld && sudo chown mysql /var/run/mysqld

sudo mysqld_safe --user=mysql --skip-grant-tables --skip-networking &

mysql -u root mysql

flush privileges;

ALTER USER  'root'@'localhost' IDENTIFIED BY 'Password=1234';

sudo killall -u mysql

sudo systemctl restart mysql.service

sudo mysql -u root -p

################## Clone DB ################

1. Make a dump of your source database:

   mysqldump -uroot -p my_project -r my_project.sql
   

   Or, if you only want to dump the database's table structure (schema) without any contents:

   mysqldump -uroot -p my_project -r my_project.sql --no-data
   

2. Open up a MySQL shell:

   mysql -uroot -p
   

3. From the MySQL shell, create a new database and populate it with the dumped data:

   CREATE DATABASE my_project_copy;
   USE my_project_copy;
   SOURCE my_project.sql;
   

4. Create a user and give it permissions to the new database:

   CREATE USER new_user IDENTIFIED BY 'some_password';
   GRANT ALL ON my_project_copy.* TO 'new_user'@'localhost' IDENTIFIED BY 'some_password';

Install TA-Lib on different OS

Ubuntu: #!/usr/bin/env bash
	## sudo apt install build-essential wget -y
	## wget https://artiya4u.keybase.pub/TA-lib/ta-lib-0.4.0-src.tar.gz
	## tar -xvf ta-lib-0.4.0-src.tar.gz
	## cd ta-lib/
	## ./configure --prefix=/usr
	## make
	## sudo make install wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz \ && sudo tar -xzf ta-lib-0.4.0-src.tar.gz \ && sudo rm ta-lib-0.4.0-src.tar.gz \ && cd ta-lib/ \ && sudo ./configure --prefix=/usr \ && sudo make \ && sudo make install \ && cd ~ \ && sudo rm -rf ta-lib/ \ && pip install ta-lib

Mac Os: ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" < /dev/null 2> /dev/null brew install ta-lib For all: pip install ta-lib

xgboost & lightgbm with gpu on windows

Install latest visual studio and R 3.5.1


XGB: 1. git clone --recursive https://github.com/dmlc/xgboost

2. cmake .. -G"Visual Studio 15 2017 Win64" -T v140 -DUSE_CUDA=ON -DR_LIB=ON

3. cmake --build . --target install --config Release

4. install.packages() with local file

LightGBM: 
1. http://lightgbm.readthedocs.io/en/latest/GPU-Windows.html
(address-model=64)
i.e., (CUDA, Rtools, Boost)

2. git clone --recursive https://github.com/Microsoft/LightGBM

cd LightGBM/R-package

3. Set use_gpu to TRUE in R-package/src/install.libs.R to enable the build with GPU support. 

4. Rscript build_package.R
  R CMD INSTALL lightgbm_2.1.1.tar.gz --no-multiarch

5. Install.packages in R with local file

Install GUI of Ubuntu on AWS

1. Connect ssh to ec2 instance.

2. install vncserver:
sudo apt install ubuntu-desktop
sudo apt install vnc4server
sudo apt install gnome-panel

3. Type 'vncserver' to setup pwd

4. Kill vncserver 'vncserver -kill :1'

5. Type following:

vim /home/ubuntu/.vnc/xstartup

and modify

unset SESSION_MANAGER
# exec /etc/.....
gnome-session -session=gnome-classic &
gnome-panel&

('ESC', ':wq', 'ENTER' to quit vim)

Type 'vncserver' to start.

Download TightVNC Viewer to connect.

In AWS website, to set "Inbound rule" to allow connections, with IP::port


Tensorflow with AWS: https://aws.amazon.com/cn/blogs/china/tensorflow-on-aws/