跳转至

首页

Filecoin 搭建

本文旨在避坑,本人作为矿工时期会保持更新。

1. 硬件与存储配置

需要的机器整体分为三种,包括主节点lotus(同步主网)、矿工节点miner()

管理节点(node200)

  • CPU Intel 4110R x 2

  • 内存128G

  • 无限卡

  • 128G系统盘(M.2)

主节点与矿工节点(node201)

  • CPU Intel 4110R x 2

  • 内存256G

  • 无显卡

工作节点1(node202)

  • CPU Intel 4110 x 2

  • 内存372G

工作节点2(node203)

  • CPU AMD 7302 x 2

  • Driver: NVIDIA-Linux-x86_64-460.91.03.run

2. 编译

我们需要根据不同的CPU型号来编译不同的二进制文件,源码文件建议放在共享目录中,如果机器数量足够多二进制文件也需要单独存放减少编译负担。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash
set -x
VERSION=v1.11.1
source /root/env-lotus
source /root/env-proxy
#curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
#wget -c https://golang.org/dl/go1.16.7.linux-amd64.tar.gz -O - | sudo tar -xz -C /usr/local
#echo "export PATH=$PATH:/usr/local/go/bin" >> ~/.bashrc && source ~/.bashrc

cd /filecoin/cache/build/lotus
make clean
git checkout master
git pull
git checkout $VERSION
git submodule deinit --all
git submodule update --init
# Some older Intel and AMD processors WITHOUT the ADX instruction support may panic with illegal instruction errors.
LSCPU=$(lscpu|grep -i adx)
if [[ $? == 0 ]]
then
    echo "Instruction ADX detected."
else
    export CGO_CFLAGS_ALLOW="-D__BLST_PORTABLE__"
    export CGO_CFLAGS="-D__BLST_PORTABLE__"
fi
# If you have an AMD Zen or Intel Ice Lake CPU (or later), ENABLE the use of SHA extensions by adding these two environment variables:
LSCPU=$(lscpu|grep -i ' sha')
if [[ $? == 0 ]]
then
    echo "Instruction SHA detected."
    export RUSTFLAGS="-C target-cpu=native -g"
    export FFI_BUILD_FROM_SOURCE=1
fi
make -j16
make lotus-bench
make install
export https_proxy=

3. 分角色配置

主节点与矿工节点(node201)

这里我们的主节点与矿工节点使用同一台主机,其环境变量配置如下。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash
export BELLMAN_CPU_UTILIZATION=0.875
export FIL_PROOFS_MAXIMIZE_CACHING=1
export FIL_PROOFS_USE_GPU_COLUMN_BUILDER=0
export FIL_PROOFS_USE_GPU_TREE_BUILDER=0
export FIL_PROOFS_USE_MULTICORE_SDR=1
export FIL_PROOFS_SDR_PARENTS_CACHE_SIZE=1073741824
#export RUST_BACKTRACE=full
#export RUST_LOG=debug

export FULLNODE_API_INFO="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.Wq0nVk1xEpwsrQhfxpk2Vb5lBS07NeJ6o4ZJRGoQuic:/ip4/192.168.0.101/tcp/1234/http"
export MINER_API_INFO="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.c6SQus7UjC4rh-OKhi45f3RGr9UyH8jURrDsA521ZQ8:/ip4/192.168.0.101/tcp/2345/http"

export LOTUS_PATH=/filecoin/data/node/ # When using a local node.
export LOTUS_MINER_PATH=/filecoin/data/miner/
export LOTUS_WORKER_PATH=/filecoin/data/worker/

export FIL_PROOFS_PARAMETER_CACHE=/filecoin/cache/parameter/
export FIL_PROOFS_PARENT_CACHE=/filecoin/cache/parent/
export TMPDIR=/filecoin/cache/tmp/


export IPFS_GATEWAY=https://proof-parameters.s3.cn-south-1.jdcloud-oss.com/ipfs/
export GOPROXY=https://goproxy.cn

3. 测试

1
编译完成后,需要对机器的能力进行简单测试,防止某些配置情况导致

3. 过程控制

虽然lotus的调度过程非常的傻,网上也有不少的文章都说对lotus的封装调度进行了优化,但是它一般是建立在机器数量较多的前提下,对于数量较少单个机器同时承载多个角色的情况,我们可以观察每个扇区在不同过程的表现来进行适当控制。

每个扇区在不同过程的消耗可以参考Task resource table。封装整体分为PreCommit与Commit两个过程,每个过程中又有两个不同阶段。其中,P1扇区可以并发,P2可以并发但会独占某个GPU,C1过程很快且可以并发,C2过程独占某个GPU且会排斥除AP外的其他任何新增阶段。

以扇区作为sectorsAgent,机器作为workerAgent,那么我们先定义sectorAgent的行为(为方便计算,假设每个扇区封装过程中占用空间为500G,不同的lotus版本、机器配置扇区封装过程表现的现象可能不同)。

1
2
3
4
5
6
sectorAgent:
AP -> P1(6h, 60G MEM) -> P2(2h, 20G MEM, GPU, Parallel) -> PreCommitAggregate -> C1(0.5h, 1G MEM) -> C2(1-2h, 200G MEM, only Parallel with GPU) -> SubmitCommitAggregate -> Submit

workerAgent:
Worker1(CPU only, 376G MEM)
Worker2(wtih GPU, 1T MEM)

软件定义无线电(SDR)的设备、软件与应用指南

注意:本文内容仅限于实验室安全测试目的,禁止用于任何商业或违反当地法律法规的活动。

不管是较贵的Ettus还是入门的HackRF,抑或是最初级的RTL-SDR设备,都可以使用这篇教程中的绝大部分内容。

GRCon2019

https://www.gnuradio.org/grcon/grcon17/presentations/

https://www.gnuradio.org/grcon/grcon18/presentations/

https://www.gnuradio.org/grcon/grcon19/presentations/

https://github.com/mossmann/hackrf/wiki

https://www.hackrf.net/hackrf%E4%B8%8Egnuradio%E5%85%A5%E9%97%A8%E6%8C%87%E5%8D%97/

http://www.hackrf.net/faq/

https://wiki.myriadrf.org/LimeSDR

https://myriadrf.org/news/limesdr-made-simple-part-1/

雪碧0xroot的PPT

0. 环境准备

0.1 硬件

HackRF One Ettus B200 Ettus B210 BladeRF x40 LimeSDR LimeSDR mini
Frequency Range 1MHz-6GHz 70MHz-6GHz 70MHz-6GHz 300MHz-3.8GHz 100kHz-3.8GHz 100kHz-3.5GHz
RF Bandwidth 20MHz 61.44MHz 61.44MHz 40MHz 61.44MHz 30.72MHz
Sample Depth 8 bits 12 bits 12 bits 12 bits 12 bits 12 bits
Sample Rate 20MSPS 61.44MSPS 61.44MSPS 40MSPS 3.2MSPS 61.44MSPS
Transmitter Channels 1 1 2 1 2 1
Receivers 1 1 2 1 2 1
Duplex Half Full Full Full Full Full
Interface USB 2.0 USB 3.0 USB 3.0 USB 3.0 USB 3.0 USB 3.0
Programmable Logic Gates 64 macrocell CPLD 75k 100k 40k (115k avail) 40k 40k
Chipset MAX5864, MAX2837, RFFC5072 AD9364 AD9361 LMS6002M LMS7002M LMS7002M
Open Source Full Schematic, Firmware Schematic, Firmware Schematic, Firmware Full Full
Oscillator Precision +/-20ppm +/-2ppm +/-2ppm +/-1ppm  +/-1ppm initial
+/-4ppm stable  +/-1ppm initial

+/-4ppm stable | | Transmit Power | -10dBm+ (15dBm @ 2.4GHz) | 10dBm+ | 10dBm+ | 6dBm |  0 to 10dBm | 0 to 10dBm | | Price | 249€ euros VAT Exc. | 991€ euros VAT Exc. | 1658€ euros VAT Exc. | 625€ euros VAT Exc. | 332€ euros VAT Exc. | 190€ euros VAT Exc. |

0.2 驱动

0.3 软件

https://unicorn.360.com/blog/2017/04/12/LimeSDR-Getting-Started-Quickly/

https://oneguyoneblog.com/2016/09/15/sdrsharp-sdr-installing-windows-10/

下载SDR#后,重启按F7进入“禁用驱动签名”的运行模式,运行其中的install-rtlsdr.bat,替换第0个驱动

1. 接收信号

接收信号建议使用gqrx(MacOS、Linux),也可以用sdrsharp(Windows)。

https://www.rtl-sdr.com/big-list-rtl-sdr-supported-software/

$ port info gqrx

$ sudo port install gqrx

接收信号以后,你可以做的内容就比较多了,这里我会举一些比较有意思的例子。

1.1. 听广播/看电视

http://dalvikplanet.blogspot.com/2017/03/how-to-get-working-rtl2832u-r820t2-on.html

1.2. 接收气象云图

SDR软件

虚拟声卡

WXtoimg

gpredict/orbitron

https://www.rtl-sdr.com/rtl-sdr-tutorial-receiving-noaa-weather-satellite-/blog/images/

https://wischu.com/archives/528.html

GSM嗅探

https://www.cnblogs.com/k1two2/p/7000942.html

1.3. 接收GPS信息

https://swling.com/blog/2016/04/guest-post-using-the-hackrf-one-for-dgps-beacon-reception/

http://sdrgps.blogspot.com/2016/12/rtl-sdr-to-orbit-with-limesdr.html

1.4. 方向探测与被动雷达 Direction Finding and Passive Rador

https://www.rtl-sdr.com/ksdr/

1.5. 接收whatever you want LEGALLY

zigbee https://github.com/bastibl/gr-ieee802-15-4

https://github.com/BastilleResearch/scapy-radio/tree/master/gnuradio/gr-zigbee

2. 发送信号

2.1. 发送GPS信号

https://gist.github.com/gyaresu/343ae51ecbb70486e270

https://www.cnblogs.com/k1two2/p/5477291.html#4245780

https://gorgias.me/2017/07/30/HackRF-GPS-%E6%AC%BA%E9%AA%97/

https://github.com/osqzss/LimeGPS

2.2. 发送文字/音视频

Windows软件sdrangel

http://gareth.codes/hackrf-transmit/

https://github.com/fsphil/hacktv

http://www.irrational.net/2014/03/02/digital-atv/

http://www.hackrf.net/2014/06/hackrf_nbfm_tx_n_ctcss_squelch/

http://www.xn--hrdin-gra.se/blog/wp-content/uploads/2015/08/nbfm-tx.grc

https://gist.github.com/gyaresu/343ae51ecbb70486e270

https://nuclearrambo.com/wordpress/transferring-a-text-file-over-the-air-with-limesdr-mini/

https://github.com/martinmarinov/TempestSDR

3. 收发信号

GSM

https://www.evilsocket.net/2016/03/31/how-to-build-your-own-rogue-gsm-bts-for-fun-and-profit/

https://yatebts.com/open_source/

https://cn0xroot.com/2017/01/10/iot-mode-fuzzing-with-openbt/

LTE

https://yq.aliyun.com/articles/310348

https://www.cnblogs.com/k1two2/p/5666667.html

https://cn0xroot.com/2017/04/12/limesdr-getting-started-quickly/

OpenBTS+LimeSDR

Prepare:

Ubuntu Desktop 16.04 & LimeSDR 1.4s with LimeSuite 17.12(If not, OpenUSRP will fail.)

Install build-essential packages

packages for soapysdr available at myriadrf PPA

sudo add-apt-repository -y ppa:myriadrf/drivers sudo apt-get update

install core library and build dependencies

sudo apt-get install -y git g++ cmake libsqlite3-dev

install hardware support dependencies

sudo apt-get install -y libsoapysdr-dev libi2c-dev libusb-1.0-0-dev

install graphics dependencies

sudo apt-get install -y libwxgtk3.0-dev freeglut3-dev

Install for building uhd

sudo apt-get install libboost-all-dev libusb-1.0-0-dev python-mako doxygen python-docutils cmake build-essential

Change to UHD driver via uhd

$ cd ~ # build and install limesuite $ git clone https://github.com/myriadrf/LimeSuite.git $ cd LimeSuite $ mkdir builddir && cd builddir $ cmake ../ $ make -j4 $ sudo make install $ sudo ldconfig

$ cd ~ # build uhd, install, enable lime, rebuild $ git clone https://github.com/EttusResearch/uhd.git $ cd uhd/host/ $ mkdir build && cd build $ cmake ../ $ make -j4 $ sudo make install $ git clone https://github.com/jocover/OpenUSRP.git lib/ursp/OpenUSRP # DO NOT GO OUT $ echo "INCLUDE_SUBDIRECTORY(OpenUSRP)">>lib/ursp/CMakeLists.txt $ cmake ../ $ make -j4 $ sudo make install

Or, Change to UHD driver via SoapySDR

$ git clone https://github.com/pothosware/SoapySDR $ cd SoapySDR $ mkdir builddir;cd builddir; cmake ../ $ make -j4 $ sudo make install

$ git clone https://github.com/myriadrf/LimeSuite $ cd LimeSuite

Build OpenBTS

4. SDR

软件定义无线电的内容即是可以灵活定义信号的处理过程,比如输出到TCP/UDP、文字音视频解码等。其中比较有名的有GNURadio、SoapySDR、Pothos(IDE)等(这里以GNURadio为例)。推荐在Linux中安装,当然也可在MacOS或者Windows中使用MacPorts进行安装,除此之外,也有PyBombs可选。

在MacOS中安装需要使用MacPorts、XQuartz,MacPorts安装内容如下。

$ port info gnuradio $ sudo port install gnuradio+wxgui gr-osmosdr sox $ port content gnuradio $ sudo port install hackrf $ sudo port install rtl-sdr $ sudo port search gr- # if you wanna more modules in gnuradio, don't be shy

然后打开XQuartz,将/opt/local/bin/gnuradio-companion加入到X自定义应用程序菜单中(建议修改默认的X终端程序内容为_xterm -e "source ~/.bash_profile;/bin/bash"_)。

https://greatscottgadgets.com/sdr/

https://gist.github.com/machinaut/addf3438ef0c1a9cad38

https://osmocom.org/projects/gr-osmosdr/wiki/GrOsmoSDR#RTL-SDRSource

https://pypi.org/project/pyrtlsdr/#description

NVIDIA Jetson使用指导

本文将以NVIDIA Jetson为硬件基础,为你展现NVIDIA的力量,可以将其作为Jetson Nano的入门参考手册(教程)。

1. 入门篇

入门篇的有两章内容,来自NVIDIA JETSON包装盒自带的内容。

1.1. 准备环境

Ref.1: https://developer.nvidia.com/embedded/learn/get-started-jetson-nano-devkit

Ref.2: https://developer.nvidia.com/embedded/downloads

这里我使用的是一个Jetson Nano开发板,起初我以为随便一个USB数据线就能把它跑起来,但是我想多了。18W的快充头插上后可以正常启动,但是一旦运行比如WebGL测试的页面直接就关机了,因而我把小米音箱的电源适配器(5V2A)给它用了,还真能跑的动。

但是为了接下来的内容,我特意买了一个5V4A的5.5mm OD的电源适配器,要不然以我现在的环境很难保证能过了接下来的准备环境阶段。

要使用这个板子,你需要提前下载SDK的SD卡镜像,以及Host OS(PC,Ubuntu 18.04)所需的SDK Manager

下载完成之后,从Ref.1的链接中下载所需的镜像烧录软件或者别的烧录软件也行,将SD卡镜像烧录至TF卡中(这里我使用的是64G TF卡,A2)。烧录完毕将其插入核心板下面的卡槽中(有些不好找),见下图。

然后通过HDMI/DP线将之与显示器连接(启动阶段的分辨率需要修改,要不然小点的屏幕没法显示启动logo,这个不是重点以后再说),插上蓝牙USB键鼠,接入USB电源,你就可以看到信仰之NVIDIA logo。

这里你需要等它初始化完成,初始化工作包括扩展跟文件系统、解压乱七八糟的包之类的,总之等看到Ubuntu Desktop的安装配置界面后可以开始操作了。

进入桌面后的第一件事儿,可以先打开Chromium,访问WebGL的示例网站,随便开个demo试试会不会关机,如果关机那么恭喜你可以找个正经的USB电源了(Ref.1里有Adafruit的USB和DC电源适配器购买链接)。

当你的DC电源到了以后,先不要直接插上,因为需要设置一下跳线,如图所示。

看到Power Jack/USB Jumper没,由于我手里没有跳线帽,所以我直接短接了它们,如图所示(看我意念焊接术)。

然后再插上刚入手的DC 5V4A,即可空出你的USB并将之与Host PC相连了。

1.2. 准备SDK

这一节你可以先跳过去,等跑完下面的小节后再看,因为这部分并不影响接下来的操作。

准备SDK的内容主要包括:下载安装Host PC、开发板所需CUDA、OpenCV之类的,需要开发板的USB连接到Host PC上作数据连接用(我没有尝试过那个USB口既作电源又作数据传输)。

这里我并不打算过多介绍,只要按照引导进行操作即可。

1.3. Hello AI World

这个示例为你充分避开了各种依赖库的复杂安装步骤以及非常多的专业术语,对新手较为友好,但是我会仍会将其以链接形式展现,在最后章节的连接中。

第一步,从Hello World开始,你仍然需要最基础的工具。

$ sudo apt install git make cmake $ git clone https://github.com/dusty-nv/jetson-inference $ cd jetson-inference $ git submodule update --init $ mkdir build $ cd build $ cmake ../ $ make -j4 $ sudo make install $ cd aarch64/bin

这一顿操作后,你会拥有个Hello AI World的全部成果。但是,what the hell an I doing?

来,对于一些Linux不熟悉的同学来说只要知道这里的cmake与make是编译源码的指令就行,cmake用来生成Makefile,make会根据Makefile里定义的动作调用gcc开始编译。

然后让我们看第一个例子,使用ImageNet的图片素材来训练我们的“机器人”让它能够识别各种物体,其中你会看到当前目录下有两个imagenet开头的文件,让我们从imagenet-console开始。

在图形界面上打开终端后,执行如下命令。

$ ./imagenet-console orange_0.jpg output_0.jpg $ nautilus .

然后经过机器人的推理以后,你会得到一张橘子、另一张还是橘子的图片,并且新橘子图片的左上角标识了机器人认为它有多大概率是橘子。

是不是有感觉了?OK,我们继续。

既然它可以看图片,那么它当然也可以看视频或者摄像头中的内容,那么接下来我们让它看到摄像头中的橘子试试。

这里我需要给Jetson Nano接一个USB摄像头,接入以后可以在终端键入cheese打开拍照应用程序看它是否工作。

然后终端中运行如下命令。

./imagenet-camera

Oops,segmentation fault了,如文档所说,默认的摄像头是板载CSI摄像头,所以这里需要修改代码让它使用后来插入的USB摄像头。

$ vi ../../../imagenet-camera/imagenet-camera.cpp

...

include "imageNet.h"

define DEFAULT_CAMERA 0 // -1 for onboard camera, or change to index of /dev/video V4L2 camera (>=0)

bool signal_recieved = false; ...

将DEFAULT_CAMERA修改为0以后,便会启用/dev/video0路径上的摄像头,然后重新编译。

$ cd ../../ # build $ cmake ../ $ make -j4

然后进到bin目录下再运行一次imagenet-camera即可。

PS:不是所有的摄像头都叫罗技C920,由于摄像头原生编码的问题,可能会导致上述程序黑屏,那么我们需要是适当修改一些内容。笔者暂时跳过这里,等改好以后再看。关于兼容列表可以参考eLinux的链接

1.4. 写一个小程序

如果你跟着github的教程,那么应该到你自己写一段代码的时间了,直接粘贴吧。

// include imageNet header for image recognition

include

// // include loadImage header for loading images

include

int main( int argc, char** argv ) { // a command line argument containing the image filename is expected, // // so make sure we have at least 2 args (the first arg is the program) if( argc < 2 ) { printf("my-recognition: expected image filename as argument\n"); printf("example usage: ./my-recognition my_image.jpg\n"); return 0; }

// retrieve the image filename from the array of command line args const char* imgFilename = argv[1]; float* imgCPU = NULL; // CPU pointer to floating-point RGBA image data float* imgCUDA = NULL; // GPU pointer to floating-point RGBA image data int imgWidth = 0; // width of the image (in pixels) int imgHeight = 0; // height of the image (in pixels)

// load the image from disk as float4 RGBA (32 bits per channel, 128 bits per pixel) if( !loadImageRGBA(imgFilename, (float4**)&imgCPU, (float4**)&imgCUDA, &imgWidth, &imgHeight) ) { printf("failed to load image '%s'\n", imgFilename); return 0; } imageNet* net = imageNet::Create(imageNet::GOOGLENET);

if( !net ) { printf("failed to load image recognition network\n"); return 0; } float confidence = 0.0; const int classIndex = net->Classify(imgCUDA, imgWidth, imgHeight, &confidence); if( classIndex >= 0 ) { // retrieve the name/description of the object class index const char* classDescription = net->GetClassDesc(classIndex); // print out the classification results printf("image is recognized as '%s' (class #%i) with %f%% confidence\n", classDescription, classIndex, confidence * 100.0f); } else { // if Classify() returned < 0, an error occurred printf("failed to classify image\n"); } delete net; // this is the end of the example! return 0; }

require CMake 2.8 or greater

cmake_minimum_required(VERSION 2.8)

declare my-recognition project

project(my-recognition)

import jetson-inference and jetson-utils packages.

note that if you didn't do "sudo make install"

while building jetson-inference, this will error.

find_package(jetson-utils) find_package(jetson-inference)

CUDA and Qt4 are required

find_package(CUDA) find_package(Qt4)

setup Qt4 for build

include(\({QT\_USE\_FILE}) add\_definitions(\))

compile the my-recognition program

cuda_add_executable(my-recognition my-recognition.cpp)

target_link_libraries(my-recognition jetson-inference)

然后编译并运行。

$ cmake . $ make $ ./my.cpp polarbear.jpg

输出结果如下。

[cuda] cudaAllocMapped 5089520 bytes, CPU 0x100c30000 GPU 0x100c30000

imageNet -- loading classification network model from: -- prototxt networks/googlenet.prototxt -- model networks/bvlc_googlenet.caffemodel -- class_labels networks/ilsvrc12_synset_words.txt -- input_blob 'data' -- output_blob 'prob' -- batch_size 2

[TRT] TensorRT version 5.0.6 [TRT] detected model format - caffe (extension '.caffemodel') [TRT] desired precision specified for GPU: FASTEST [TRT] requested fasted precision for device GPU without providing valid calibrator, disabling INT8 [TRT] native precisions detected for GPU: FP32, FP16 [TRT] selecting fastest native precision for GPU: FP16 [TRT] attempting to open engine cache file /usr/local/bin/networks/bvlc_googlenet.caffemodel.2.1.GPU.FP16.engine [TRT] loading network profile from engine cache... /usr/local/bin/networks/bvlc_googlenet.caffemodel.2.1.GPU.FP16.engine [TRT] device GPU, /usr/local/bin/networks/bvlc_googlenet.caffemodel loaded [TRT] device GPU, CUDA engine context initialized with 2 bindings [TRT] binding -- index 0 -- name 'data' -- type FP32 -- in/out INPUT -- # dims 3 -- dim #0 3 (CHANNEL) -- dim #1 224 (SPATIAL) -- dim #2 224 (SPATIAL) [TRT] binding -- index 1 -- name 'prob' -- type FP32 -- in/out OUTPUT -- # dims 3 -- dim #0 1000 (CHANNEL) -- dim #1 1 (SPATIAL) -- dim #2 1 (SPATIAL) [TRT] binding to input 0 data binding index: 0 [TRT] binding to input 0 data dims (b=2 c=3 h=224 w=224) size=1204224 [cuda] cudaAllocMapped 1204224 bytes, CPU 0x101310000 GPU 0x101310000 [TRT] binding to output 0 prob binding index: 1 [TRT] binding to output 0 prob dims (b=2 c=1000 h=1 w=1) size=8000 [cuda] cudaAllocMapped 8000 bytes, CPU 0x101440000 GPU 0x101440000 device GPU, /usr/local/bin/networks/bvlc_googlenet.caffemodel initialized. [TRT] networks/bvlc_googlenet.caffemodel loaded imageNet -- loaded 1000 class info entries networks/bvlc_googlenet.caffemodel initialized. class 0296 - 1.000000 (ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus) image is recognized as 'ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus' (class #296) with 100.000000% confidence

简言之,程序使用ImageNet的图片配上GoogleNet的模型对你的图片进行推理,然后得出它认为这是北极熊的可能性。

因为这篇文章的目的是入门,也就是带进来以后看哪个方向就自己看,所以相关数学知识在这里已经忽略了,如果你能写出厉害的理论paper又做出很厉害的工程实现,那么大牛就请继续往下过,顺便留个言交个朋友让我膜拜一下。你也可以查看文末的链接进一步扩展阅读什么是GoogleNet,什么是CNN,然后撸一遍机器学习、深度学习、强化学习啥的,也可能一路懵懂复习到信号与系统、高等数学,你要书的话我这还卖,另外我的学习笔记可以参阅DataNote

2. 进阶篇

2.1. 重新训练模型

2.2. 作为推理节点

2.3. 深度学习实验

2.4. TensorFlow实验

安装TensorFlow

$ sudo apt-get install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev $ sudo apt-get install python3-pip $ sudo pip3 install -U pip $ sudo pip3 install -U numpy grpcio absl-py py-cpuinfo psutil portpicker six mock requests gast h5py astor termcolor protobuf keras-applications keras-preprocessing wrapt google-pasta $ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v42 tensorflow-gpu==1.14.0+nv19.9

https://devtalk.nvidia.com/default/topic/1048776/official-tensorflow-for-jetson-nano-/

https://www.tensorflow.org/tutorials

with jupyter

2.5. TensorRT实验

3. 行业方案篇

太阳底下没有新东西,我发现把之前的笔记稍微整理点可以新开一个目录出一个系列,那么,这里我就直接写关键字吧,以后说不定又冒出什么新东西了呢。

在继续之前,我们需要抓住一样内容,即凡是人类自己通过观察、模仿、学习可以获得的重复能力,机器/深度/强化等内容理论来说都可以帮你实现,即使是创造力。

然后我们再讲方案,即在这个小小的盒子,它有多少能力,能干啥。

3.1. 边缘计算

工业现场

计算转移

CDN

军用头戴

3.2. 云游戏

4K/8K/HDR/60FPS家用主机、服务端

无主机头戴

3.3. 教学设备

甭管便宜的贵的只要带卡都能当教学设备,不信你看研究生论文有多少CUDA相关。

3.4. 残障辅助

手语翻译(https://github.com/EvilPort2/Sign-Language),可按照中国残联手语进行训练(http://www.cdpf.org.cn/special/zgsy/node_305701.htm)。

apt install python3-keras

辅助视觉(物体/人脸识别后转语音)

参考:

文献

[1] NVIDIA AI Two Days Demo: https://developer.nvidia.com/embedded/twodaystoademo

[2] CNN Architecture: https://medium.com/@sidereal/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5

[3] eLinux: https://www.elinux.org/Jetson

术语

[1] ImageNet: http://image-net.org/download

[2] TensorRT: https://developer.nvidia.com/tensorrt

[3] DeepStream: https://developer.nvidia.com/deepstream-sdk

[4] cuDNN

[5] PyTorch

[6] NVDLA

title: "向NVIDIA Jetson Nano中移植QEMU-KVM" date: 2019-04-14 categories: - "cloud-infra" - "devices"


因为Jetson如果作为边缘设备,那么我们需要进一步探索虚拟化在其上的可能性,从而使FT有更容易的路线可走,还有既然它的芯片是PCIE的,那理应可以透传。

参考:https://elinux.org/Jetson/Nano/Upstream

什么build rootfs、uboot之类的就不要了,那是后期嵌入式的活,我们在现有环境上build kernel即可。

1. 准备环境

访问链接https://developer.nvidia.com/embedded/downloads并下载源码包,包括Jetson自有以及L4T源码,也可以点击如下链接直接下载。

https://developer.nvidia.com/embedded/dlc/l4t-sources-32-1-jetson-nano

解压其中的kernel部分。

https://developer.nvidia.com/embedded/dlc/l4t-jetson-driver-package-32-1-jetson-nano

下载并解压后,得到如下文件系统。

2. 准备kernel

Host: sudo su sudo apt install nfs-kernel-server sudo echo "/home/lofyer/Downloads *(rw,no_root_squash,no_subtree_check)" >> /etc/exports sudo exportfs -avf

Jetson Nano: sudo su apt instlal libncurses-dev mount root@192.168.0.59:/home/lofyer/Downloads /mnt cd /mnt/ cp /proc/config.gz . gunzip config.gz mv config .config make menuconfig # find and enable kvm, tegra hypervisor make -j4; make -j4 modules_install make -j4 Image cp arch/arm64/boot/Image /boot/Image-kvm

然后编辑启动项,默认从新kernel启动。

vi /boot/extlinux/extlinux.conf

TIMEOUT 10 DEFAULT secondary

MENU TITLE p3450-porg eMMC boot options

LABEL primary MENU LABEL primary kernel LINUX /boot/Image INITRD /boot/initrd APPEND ${cbootargs} rootfstype=ext4 root=/dev/mmcblk0p1 rw rootwait

LABEL secondary MENU LABEL kernel with kvm LINUX /boot/Image-kvm INITRD /boot/initrd APPEND ${cbootargs} rootfstype=ext4 root=/dev/mmcblk0p1 rw rootwait

3. 尝试qemu-kvm

自带的:

apt install qemu-kvm kvm --help

自己编的:

git clone https://github.com/qemu/qemu cd qemu ./configure --enable-kvm make -j4

只能使用machine类型为arm进行加速。

4. 看看FT

算了,现在不看了,等下半年。

title: "基于ARM(NVIDIA-JETSON-NANO)编译NetLogo" date: 2019-04-06 categories: - "abm" - "devices" -tags: - "NVIDIA" - "Jetson Nano" - "NetLogo"


本文主要目的为测试这块板子的性能,看看其是否有作为边缘节点的能力。

Env

Ubuntu 18.04(jetson-nano-sd-r32.1-2019-03-18)

Prepare

$ sudo set -i 's/ports.ubuntu.com/mirrors.ustc.edu.cn/g' /etc/apt/sources.list $ sudo apt install -y curl x11vnc openjdk openjfx libopenjfx-jni libopenjfx-java

Build

$ git clone https://github.com/NetLogo/NetLogo $ cd NetLogo $ git submodule update --init $ ./sbt netlogo/compile

Run and Package

$ ./sbt netlogo/run $ ./sbt dist/buildNetLogo

Addon: Add VNC server to your board

$ sudo apt install -y x11nvc $ x11vnc # to generate ~/.vnc files $ echo 'x11vnc --loop &' >> ~/.xsessionrc

Ref

https://github.com/NetLogo/NetLogo/wiki/Building

https://github.com/NetLogo/NetLogo/wiki/Releasing

这是一篇表达

距离上一次写已经很久了,而且似乎有个趋势就是对生活越熟悉就越麻木,也同样伴随体重的增加。

写了一句话,我已经忘了落笔时的念想。

要偶尔转移压力,但不是通过思考,方法或许成熟,但是经不得劳累。

要放弃linode的集群了,因为vultr ipv6内部通信更便宜。

HPC相关

HPC入门指南(2018-08-13)

1. History

Wikipedia is good. https://en.wikipedia.org/wiki/Supercomputer

2. Stack

2.1. Hardware

Storage Network Server Cooler

2.2. Software

Infrastructure deployment Parallel Filesystem Workload Management Development

Finally, we've got a PPT.

但是这个PPT很长,我只能截一些出来。

HPC增补(2022-02)

HPC的生意目前来看还是集成,部分组件可以标准化,但是克雷、富士通、IBM、NVIDIA都有自己的盘子,库也都是为自家架构优化,这些都没有问题,理解。

想搞高性能向HPC技术方向学习,也没有问题,但是搞完过后想复制,那就铺人天了。一堆定制设备、一堆专有技术栈,加上已知最好的设备和软件都是国外垄断,生意做着做着就自己就成集成商了。集成商没啥不好,毕竟150W刀一个P的算力,这生意走量。国产化版本堆算力早些年成本差不多两倍,但现在差距在逐步缩小。

HPC的生意不好做,进不去圈子,但是汤可以喝,而且大厂们也想喝这个汤,不信你看GPFS都卖给了谁。后来做的纯软裸金属算是产品入局了,但是再往上走适配一个已经很很很完整的生态,想想都觉得累。

算了,回头把RDMA技术栈补一补,这样或许可以再搞些其他的花样,只是产品化的路没那么好走,但是集成的生意要啥产品化呢,还是要的,你看SAP模块化也厉害的一塌糊涂。

title: "构建基于ARM的超算集群" date: 2019-08-26 categories: - "cloud-infra" - "devices" - "hpc"


Background

Nowadays ARM-based servers are being applied to many scenarios, so we are going to port some workload to the minimal ARM SoC with NVIDIA GPU to evaluate the feasibility of production usage.

Preparation

Hardware: Legacy ARM SoC boards, e.g. Raspberry Pi, Beagle Board, NVIDIA Jetson(GPU) and standard ARM-based servers.

MPI: OpenMPICH and MPICH2.

Share Storage: ARM-based Glusterfs with pNFS.

Application: ANY

Build/Installation

Ref:

[1] https://developer.arm.com/solutions/hpc

fpga-ft

"基于FPGA的物理机容错(FT)或者虚拟机容错设计" "2019-05-26"

1. Background

Since the MicroBlaze provides the lockstep feature, so finally we can make a POC.

2. MicroBlaze FT Test

3. Server FT Test via MicroBlaze FT controller

4. QEMU with COLO

[1] Triple Modular Redundancy: https://www.xilinx.com/support/documentation/ip_documentation/tmr/v1_0/pg268-tmr.pdf

[2] Fault Tolerance Technique for Dynamically Reconfigurable Processor: https://pdfs.semanticscholar.org/2e98/b34ee8736eba7747b223c333de5739a6e601.pdf

[3] Xilinx Reduces Risk and Increases Efficiency for IEC61508 and ISO26262 Certified Safety Applications: https://www.xilinx.com/support/documentation/white_papers/wp461-functional-safety.pdf

[4] Spartan-6 FPGA Dual-Lockstep MicroBlaze Processor with Isolation Design Flow: https://www.xilinx.com/support/documentation/application_notes/xapp584-dual-lockstep-microblaze-IDF.pdf

[5] MicroBlaze Processor Reference Guide: https://www.xilinx.com/support/documentation/sw_manuals/xilinx2019_2/ug984-vivado-microblaze-ref.pdf

fpga-test

"PetaLinux/Pynq中使用QEMU(裸金属探索的路子)" "2019-08-22"

Background

Before stepping into the virtio acceleration, we will build and run a Linux(PetaLinux) in FPGA.

Despite the poor performance of simulation in X86, it's very convenient to create POC.

We choose MPSoC zcu106(PS) to test ARM, Zynq zc702(PS) to test ARM and Kintex UltraScale kcu705(PL) to test MicroBlaze softcore.

0. Essential tools

OS: Ubuntu 16.04 Desktop

Vivado: 2019.1

Xilinx SDK: 2019.1

PetalLinux SDK:  PetaLinux 2019.1

BSP: Evaluation Board BSP or generated from Vivado/Xilinx SDK

1. Build QEMU

QEMU is in the PetaLinux SDK directory, it's not necessary to rebuild if you do not wanna modify anything.

git clone git://github.com/Xilinx/qemu.git cd qemu apt install libglib2.0-dev libgcrypt20-dev zlib1g-dev autoconf automake libtool bison flex libpixman-1-dev git submodule update --init dtc ./configure --target-list="aarch64-softmmu,microblazeel-softmmu" --enable-fdt --disable-kvm --disable-xen make -j4

2. PetaLinux Project

2.1. Create Project

petalinux-create -t project -n zcu106_arm_a53 --template zynqMP -s ../bsp/xilinx-zcu106-v2019.1-final.bsp petalinux-config # Decide what to build

After the project created, pre-built images is in directory pre-built. You can test it via petalinux-boot

petalinux-boot --qemu --prebuilt 3 # Start ZCU106 virtual machine with prebuilt kernel

To accelerate the sstate mirror check process, you need download the sstate cache in Ref [5].

If you are using petalinux as root, you need modify sanity.conf like this:

cat /opt/petalinux/v2019.1/components/yocto/source/aarch64/layers/core/meta/conf/sanity.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Sanity checks for common user misconfigurations
#
# See sanity.bbclass
#
# Expert users can confirm their sanity with "touch conf/sanity.conf"
BB\_MIN\_VERSION = "1.39.1"

SANITY\_ABIFILE = "${TMPDIR}/abi\_version"

SANITY\_VERSION ?= "1"
LOCALCONF\_VERSION  ?= "1"
LAYER\_CONF\_VERSION ?= "7"
SITE\_CONF\_VERSION  ?= "1"

#INHERIT += "sanity"
2.2. Build Kernel/Rootfs

3. Boot from QEMU

3.1. Debug

petalinux-boot --qemu --prebuilt 3

3.2. Run with self-compiled-QEMU

!/bin/bash

/root/Desktop/qemu_zynq_devices/qemu/aarch64-softmmu/qemu-system-aarch64 \ -M arm-generic-fdt-7series -machine linux=on -smp 2 -m 1G \ -serial /dev/null -serial mon:stdio -display none \ -kernel /blog/images/linux/uImage -dtb /blog/images/linux/system.dtb

Now we'll add sdcard and try to boot Ubuntu Core OS.

qemu-img create qemu_sd.img 2G -M arm-generic-fdt-7series -machine linux=on -smp 2 -m 1G \ -serial /dev/null -serial mon:stdio -display none \ -kernel /blog/images/linux/uImage -dtb /blog/images/linux/system.dtb

4. Co-Sim

5. Boot from Evaluation Board SDCard

Ref:

[1] Build QEMU: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842060/QEMU

[2] Zynq UltraScale+ QEMU: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841606/QEMU+-+Zynq+UltraScalePlus

[3] CoSim: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18842109/QEMU+SystemC+and+TLM+CoSimulation

[4] Xilinx Evaluation Board BSP files and Yocto local mirror: https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-design-tools.html


"Xilinx Pynq的overlay网络测试" "2020-04-14"

1. Preparation

https://github.com/Xilinx/PYNQ-Networking

2. Build

备份工具(迁云必备)

直接说结论吧,使用下来,UrBackup与Duplicati是较为合适的两个选择。

UrBackup采用传统的CS架构,对全平台提供支持,备份选项较为丰富,而且支持公网中继; Duplicati则是较为轻量级的选择,只要安装agent即可,支持备份目标地址较为丰富,除了传统接口外也支持AWS S3接口; Bacula与BareOS(前者变种)属于同一体系,但后者支持更好广泛些; Amanda除了硬盘文件备份,也多了一个MySQL数据库的备份产品; 最后一个KVMBackup则是一个脚本,限制比较多,适合单纯的KVM环境备份。

如果与ZStack对接,这俩都可以,备份存储地址可以选择FTP,这样就可以直接使用ZStack的ZStore镜像存储,格式使用raw,然后就直接启动了。

外汇学习笔记

早先把书的内容过了一遍,现在打算把书中的模型统统实现并做相应介绍。

Agent based modeling Strategy in forex trading market

PREFACE

Since I've tested S1, S2, and S3 from the DATANOTE. However, it was not so good or in other words, identical with expectation. It's time to make a change.

BASIC MODEL AND ARCHITECTURE

As you can see from my last articles, the architecture's infrastructure is about to go and the agent's protocol is well defined. Now what we should do is to combine them together.

In the first phase, the simulation will JUST be a Netlogo program. If the performance is just like S1-3, then it goes to phase 2 ---- treating the agents as the real from the data gained and strategy trained from the real world. And if NOT, less probable, the program must be corrected.

PRACTICAL GUIDE

1. FAKE WORLD, FAKE AGENT
2. REAL WORLD, FAKE AGENT
3. REAL WORLD, REAL AGENT

World3 Model is a choice.(System Dynamics)

https://insightmaker.com/insight/1954/The-World3-Model-A-Detailed-World-Forecaster

title: "Forex data collection and backtesting with python/netlogo" date: 2017-03-13 categories: - "signal-system"


These two sites provides free forex tick/time data. So before getting into backtesting, we must get data.

I provide two scripts for you to download and update these data daily.

OK, it's boring...

http://github.com/lofyer/forex-data

However, there's still some work to update our data every day/week.

Like this:

TBD

For the next, we'll try to apply basic strategy to backtest our data.

Like this:

Forex Data ML Relationship

Data Processing with ML

Data Overview: 1. Forex Index <-> Currency Rate Change 2. News -> Classification -> Weight <-> Currency Rate Change

Caution: Strict Time Based Data pipeline.

外汇常用技术指标算法(附Python量化分析教程)

公式不好写,写到DataNote里吧。

Bill Williams Indicators

Accelerator/Decelerator oscillator(AC)

Acceleration/Deceleration (AC) technical indicator signals the acceleration or deceleration of the current market driving force.

How to Use Acceleration/Deceleration

The indicator is fluctuating around a median 0.00 (zero) level which corresponds to a relative balance of the market driving force with the acceleration. Positive values signal a growing bullish trend, while negative values may be qualified as a bearish trend development. The indicator changes its direction before any actual trend reversals take place in the market therefore it serves as an early warning sign of probable trend direction changes. The indicator color changes alone are very important as the current green line (irrespectively of the value) warns against long positions the same way as the red one would against selling. To enter the market along with its driving force one needs to watch for both value and color. Two consecutive green columns above the zero level would suggest you to enter the market with a long position while at least two red columns below the zero level might be taken for a command to go short. Fake signals prevail in timeframes smaller than 4 hours.

Acceleration/Deceleration Oscillator Formula (Calculation)

AC bar chart is the difference between the value of 5/34 of the driving force bar chart and 5-period simple moving average, taken from that bar chart.

AO = SMA(median price, 5)-SMA(median price, 34) AC = AO-SMA(AO, 5)

--------------------------------------------------------------------------- Alligator

Alligator is an indicator designed to signal a trend absence, formation and direction. Bill Williams saw the alligator's behavior as an allegory of the market's one: the resting phase is turning into the price-hunting as the alligator awakes so that to come back to sleep after the feeding is over. The longer the alligator is sleeping the hungrier it gets and the stronger the market move will be.

How to Use Alligator Indicator

The Alligator indicator consists of 13-, 8- and 5-period smoothed moving averages shifted into the future by 8, 5 and 3 bars respectively which are colored blue, red and green representing the alligator’s jaw, teeth and lips. The Alligator is resting when the three averages are twisted together progressing in a narrow range. The more distant the averages become, the sooner the price move will happen. The averages continuing in an upward direction (green followed by red and blue) suggest an emerging uptrend which we interpret as a signal to buy. The averages following each other in the reversed order down the slope are a strong signal of an unfolding downtrend so selling at this point would be more than appropriate.

Alligator Indicator Formula (Calculation)

MEDIAN PRICE = (HIGH + LOW) / 2 ALLIGATORS JAW = SMMA (MEDEAN PRICE, 13, 8) ALLIGATORS TEETH = SMMA (MEDEAN PRICE, 8, 5) ALLIGATORS LIPS = SMMA (MEDEAN PRICE, 5, 3)

--------------------------------------------------------------------------- Awesome oscillator(AO)

Awesome Oscillator (AO) is a momentum indicator reflecting the precise changes in the market driving force which helps to identify the trend’s strength up to the points of formation and reversal.

How to Use Awesome Oscillator

There are three main signals which may be seen: Saucer - three consecutive columns above the nought line the first two of which must be colored red (the second one is lower than the first one) while the third one is colored green and higher than the previous (second) one. Such a formation would be a clear Buy signal whilst inverted and vertically flipped formation would serve as a Sell signal. Nought line crossing - the histogram crosses the naught line in an upward direction changing its values from negative to that of positive ones. In this situation we have a Buy signal. The Sell signal would be a reversed pattern. Two pikes - the indicator displays a Buy signal when the figure is formed by two consecutive pikes both of which are below the naught line and the later-formed pike is closer to the zero level than the earlier-formed one. The Sell signal would be given by the reverse formation.

Awesome Oscillator Trading Strategy

Awesome Oscillator Strategy includes 3 ways of trading. The first way is to open a sell position when the oscillator is below the zero line forming a peak, and open a buy position when the oscillator is above the zero line forming a gap. Another way is to open a sell position when the oscillator forms two peaks above the zero line, where the second high is lower than the previous one. And, conversely, traders watch to open a buy position when the oscillator forms to lows below the zero line with the last one not as low as the previous one. The third way is to account crossing the zero line. When the oscillator crosses it from up to down, it is time to open a sell position and when it crosses from down to up, it is time to open a buy position.

Awesome Oscillator Formula (Calculation)

Awesome Oscillator is a 34-period simple moving average, plotted through the central points of the bars (H+L)/2, and subtracted from the 5-period simple moving average, graphed across the central points of the bars (H+L)/2. MEDIAN PRICE = (HIGH+LOW)/2 AO = SMA(MEDIAN PRICE, 5)-SMA(MEDIAN PRICE, 34) where SMA — Simple Moving Average.

--------------------------------------------------------------------------- Fractal Indicator Definition

Fractals is an indicator highlighting the chart’s local heights and lows where the price movement had stopped and reversed. These reversal points are called respectively Highs and Lows. How to Use Fractal Indicator

Bill Williams' Fractals are formed around a group of five consecutive bars the first two of which are successively reaching higher (or diving deeper) and the last two descending lower (or growing higher) with the middle one being the highest (or the lowest) result in the group accordingly. Buy fractal is an arrow pointing to the top Sell fractal is an arrow pointing to the bottom -------------------------------------------------------------------------- Gator Oscillator Definition

The Gator Oscillator (GO) is a supplement to the Alligator indicator and is used alongside with it showing the absolute degree of convergence/divergence of the Alligator's three SMAs pointing at the Alligator's periods of slumber and awakeness (i.e. trending and non-trending market phases). How to Use Gator Oscillator

Being an oscillator in the form of two histograms built on either side of the naught line, the Gator Oscillator plots the absolute difference between the Alligator’s Jaw and Teeth (blue and red lines) in the positive area and the absolute difference between the Alligator’s Teeth and Lips (red and green lines) in the negative area. The histogram’s bars are colored green if exceeding the previous bar’s volume or red if falling short. The bars of the extreme values are in tune with the strong trend forces. The Alligator's activity periods are divided into the following four: Gator awakes - the bars on different sides of the naught line are colored differently. Gator eats - green bars on both sides of the naught line. Gator fills out - single red bar during the "eating" phase. Gator sleeps - the bars on both sides are red -------------------------------------------------------------------------- Market Facilitation Index Definition

The Market Facilitation Index is designed for evaluation the willingness of the market to move the price. The indicator's absolute values alone cannot provide any trading signals unlike it's dynamics in relation to the dynamics of the volume. How to Use Market Facilitation Index

The absolute values of the index are represented by the histogram's bars while the comparison of the index and volume dynamics are given in colors which are vital in terms of reading the indicator signs. Green bar - both MFI and volume are up. Increasing trading activity means market movement acceleration. We may join the trend. Blue bar - MFI indicator is up, volume is down. The movement is continuing although the volume has dropped. The trend will soon be reversing. Pink bar - MFI indicator is down, volume is up. The slowing down movement while volume is raising may indicate a possible break through, often a U-turn. Brown bar - both MFI and volume are down. The market is no longer interested in the current direction and is looking for signs of a future development. Market Facilitation Index Market Facilitation Index (BW MFI) Market Facilitation Index Formula (Calculation)

BW MFI = (HIGH-LOW)/VOLUME --------------------------------------------------------------------------

Oscilators

Trend Indicators

Volume Indicators

Accumulation/Distribution(AD)

Accumulation/Distribution is a volume-based technical analysis indicator designed to reflect cumulative inflows and outflows of money for an asset by comparing close prices with highs and lows and weighting the relation by trading volumes.

How to Use Accumulation/Distribution

The Accumulation/Distribution line is used for trend confirmation or possible turning points identification purposes. Trend confirmation: An uptrend in prices is confirmed if A/D line is rising; A downtrend in prices is confirmed if A/D line is falling. Divergence pattern analysis: Rising A/D line along with decreasing prices indicates the downtrend may be weakening to a bullish reversal; Falling A/D along with rising prices indicates the uptrend may be weakening to a bearish reversal.

Accumulation/Distribution Indicator Formula (Calculation)

A/D(t) = [((C – L) – (H – C)) / (H – L)] x Vol + A/D(t-1), where: A/D(t) – current Accumulation/Distribution value; A/D(t-1) – previous Accumulation/Distribution value; H – current high; L – current low; C – close price; Vol – volume.

---------------------------------------------------------------------------

Average Directional Movement Index(ADX)

Average Directional Index (ADX) is a technical indicator developed by Welles Wilder to estimate trend strength and determine probable further price movements by comparing the difference between two consecutive lows with the difference between the highs.

How to Use ADX Indicator

ADX is a complex indicator, which results from calculation of the Plus Directional Indicator (+DI – green line) and the Minus Directional Indicator (-DI – red line), but all of them may be used for trend analysis.

In general the indicator (bold line) move is believed to reflect current trend strength: Rising ADX (usually climbing above 25) suggests strengthening market trend – trend following indicators are becoming more useful; Falling ADX suggests the trend development is doubtful. ADX values below 20 may indicate neutral trend is present – oscillators are becoming more useful. Use of complex ADX trading system may require additional confirmation signals: Normally if +DI (green line) climbs above -DI (red line), a buy signal is generated; Normally if -DI climbs above +DI, a sell signal is generated. Average Directional Index Indicator - ADX Indicator Average Directional Index (ADX) Indicator

ADX Trading Strategy

ADX trading strategy aims to identify the strongest trends and distinguish between trending and non-trending conditions. ADX reading above 25 indicates trend strength, while when ADX is below 25, this shows trend weakness. Breakouts, which are not difficult to spot, also help to identify whether ADX is strong enough for the price to trend or not. Thus, when ADX rises from below 25 to above 25, trend is considered strong enough to continue in the direction of the breakout. It’s a common misperception that when ADX line starts falling this is a sign of trend reversal. Whereas, it only means that the trend strength is weakening. As long as ADX is above 25, it should be considered that a falling ADX line is simply less strong.

ADX Formula (Calculation)

ADX = MA [((+DI) – (-DI)) / ((+DI) + (-DI))] x 100; where: +DI – Plus Directional Indicator; -DI – Minus Directional Indicator.

---------------------------------------------------------------------------

Average True Range(ATR) --------------------------------------------------------------------------- Bears Power --------------------------------------------------------------------------- Bollinger Bands --------------------------------------------------------------------------- Bulls Power --------------------------------------------------------------------------- Commodity Channel Index(CCI) --------------------------------------------------------------------------- DeMarker --------------------------------------------------------------------------- Envelopes --------------------------------------------------------------------------- Force --------------------------------------------------------------------------- Fractals --------------------------------------------------------------------------- Gator oscillator --------------------------------------------------------------------------- Ichimoku Kinko Hyo --------------------------------------------------------------------------- Market Facilitation Index(BWMFI) --------------------------------------------------------------------------- Momentum --------------------------------------------------------------------------- Money Flow Index --------------------------------------------------------------------------- 移动平均线(Moving Average,MA)

MA=N日收市价之和/N 一般N可取短期5或10天,也可是30、65、200、280等中长期。

Moving Average is a technical analysis tool that shows average price over a given period of time, which is used to smoothen price fluctuations and therefore to determine trend direction and strength. Depending of the method of averaging, distinguish between simple moving average (SMA), smoothed moving average (SMMA) and exponential moving average (EMA).

Moving Average Trading Strategy

Moving average strategy is essentially a trend following means. Its objective is to signal the beginning of a new trend or a trend reversal. Herein, its main purpose is to track the progress of the trend and not to predict market action in the same sense that technical analysis attempts to do. By its nature, Moving Average is a follower; it follows the market telling that a new trend has begun or reversed only after the fact.

Moving Average Formula (Calculation)

SMA = Sum (Close (i), N) / N, where: Close (i) – current close price; N – period of averaging. EMA(t) = EMA(t-1) + (K x [Close(t) – EMA(t-1)]), where: t – current period; K = 2 / (N + 1), N – period of averaging.

How to Use Moving Average

Generally moving average curves analysis includes the following principles: Direction of moving average curve reflects prevailing trend over a period; Low-period averaging may give more false signals, while large-period averaging tend to be lagging; To increase (decrease) sensitivity of the curve one should decrease (increase) the period of averaging; Average curves are more useful in trending environment.

Comparing moving average with price movements: A strong buy (sell) signal arise if price crosses from below (from above) its rising (falling) moving average curve; A weak buy (sell) signal arise if price crosses from below (from above) its falling (rising) moving average curve. Comparing moving average curves of different periods: A rising (falling) lower-period curve crossing from below (above) another rising (falling) longer-period curve gives a strong buy (sell) signal; A rising (falling) lower-period curve crossing from below (above) another falling (rising) longer-period curve gives a weak buy (sell) signal. --------------------------------------------------------------------------- Moving Average of Oscillator --------------------------------------------------------------------------- 抛物线转向(Stop And Reverse)

SAR(n)=SAR(n-1)+AF*[EP(n-1)-SAR(n-1)]

其中:SAR(n)为第n日的SAR值,SAR(n-1)为第(n-1)日的值;AF为加速因子(或叫加速系数),EP为极点价(最高价或最低价)。

-*- coding: utf-8 -*-

""" http://virtualizedfrog.wordpress.com/ 2014

Translated from http://www.amibroker.com/library/detail.php?id=268 Requires pandas to load csv files, and matplotlib to chart the data

The main expects table.csv file. Valid files can be downloaded on Yahoo Finance eg: http://real-chart.finance.yahoo.com/table.csv?s=%5EGSPC&ignore=.csv """

import pandas as pd from datetime import datetime import matplotlib.pyplot as plt

def psar(barsdata, iaf = 0.02, maxaf = 0.2): length = len(barsdata) dates = list(barsdata['Date']) high = list(barsdata['High']) low = list(barsdata['Low']) close = list(barsdata['Close']) psar = close[0:len(close)] psarbull = [None] * length psarbear = [None] * length bull = True af = iaf ep = low[0] hp = high[0] lp = low[0]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
for i in range(2,length):
    if bull:
        psar\[i\] = psar\[i - 1\] + af \* (hp - psar\[i - 1\])
    else:
        psar\[i\] = psar\[i - 1\] + af \* (lp - psar\[i - 1\])

    reverse = False

    if bull:
        if low\[i\] < psar\[i\]:
            bull = False
            reverse = True
            psar\[i\] = hp
            lp = low\[i\]
            af = iaf
    else:
        if high\[i\] > psar\[i\]:
            bull = True
            reverse = True
            psar\[i\] = lp
            hp = high\[i\]
            af = iaf

    if not reverse:
        if bull:
            if high\[i\] > hp:
                hp = high\[i\]
                af = min(af + iaf, maxaf)
            if low\[i - 1\] < psar\[i\]:
                psar\[i\] = low\[i - 1\]
            if low\[i - 2\] < psar\[i\]:
                psar\[i\] = low\[i - 2\]
        else:
            if low\[i\] < lp:
                lp = low\[i\]
                af = min(af + iaf, maxaf)
            if high\[i - 1\] > psar\[i\]:
                psar\[i\] = high\[i - 1\]
            if high\[i - 2\] > psar\[i\]:
                psar\[i\] = high\[i - 2\]

    if bull:
        psarbull\[i\] = psar\[i\]
    else:
        psarbear\[i\] = psar\[i\]

return {dates:dates, high:high, low:low, close:close, psar:psar, psarbear:psarbear, psarbull:psarbull}

if __name__ == "__main__": import sys import os

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
if len(sys.argv) < 2:
    sys.exit("Usage: %s datafile.csv" % sys.argv\[0\])
if not os.path.exists(sys.argv\[1\]):
    sys.exit("Error: can't open file '%s': No such file" % sys.argv\[1\])

barsdata = pd.read\_csv(sys.argv\[1\])
#Reindex the data: ascending dates are expected in the function
barsdata = barsdata.reindex(index=barsdata.index\[::-1\])
#Convert strings to actual timestamps
barsdata\['Date'\] = \[datetime.strptime(x, '%Y-%m-%d') for x in barsdata\['Date'\]\]

startidx = 0
endidx = len(barsdata)

result = psar(barsdata)
dates = result\['dates'\]\[startidx:endidx\]
close = result\['close'\]\[startidx:endidx\]
psarbear = result\['psarbear'\]\[startidx:endidx\]
psarbull = result\['psarbull'\]\[startidx:endidx\]

plt.plot(dates, close)
plt.plot(dates, psarbull)
plt.plot(dates, psarbear)
plt.grid()
plt.show()

动向指标(DMI) --------------------------------------------------------------------------- Moving Averages Convergence/Divergence(MACD) --------------------------------------------------------------------------- On Balance Volume(OBV) --------------------------------------------------------------------------- 相对强弱指标(Relative Strength Index,RSI) --------------------------------------------------------------------------- Relative Vigor Index(RVI) --------------------------------------------------------------------------- 漫步随机指标(Random Walk Index,RWI) --------------------------------------------------------------------------- Standard Deviation(StdDev) --------------------------------------------------------------------------- Stochastic Oscillator --------------------------------------------------------------------------- Larry Williams' Percent Range(WPR) --------------------------------------------------------------------------- Python量化分析中文

Get ready for forex

这个月书的样章要完结,还有一篇稿要写。

偶然看到高频交易,加上之前对股市的自动交易感兴趣,然后偶然看到外汇交易,metatrader,看到他们的交易策略,我就燃了。(亏)新手轻虐。

Scott E.Page是个好老师。

节奏要加快,再快,再快点。

基于外汇市场技术指标的ABM建模

本模型的最终模拟期望是随机,我想试试。

货币对作为agent,这样仅需要个位数以内;

货币对本身不拥有市场价值,初始值为此货币对相对于eurusd的比值;

agent拥有自己的技术指标,这个指标尽量在一定范围内震荡;

agent行动时,随之变化的指标会影响其他agent的行动;

patch横向分层,高中低区,agent的行动只在patch间行动;

云计算笔记


"Use dhcp or dnsmasq as DHCP server in libvirt-based environment" "2015-08-26"

Default dhcp and dnsmasq are conflict in a OS. Just change to another if one failed.

1
# yum install dhcp

/etc/sysconfig/dhcpd is no longer needed in RHEL-7. However, it is still working in other distributions. Just assign an IP to the interface which in the subnet you want dhcp serve.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[root@localhost ~]# vi /etc/sysconfig/dhcpd
DHCPDARGS="eth0 eth1";

[root@localhost ~]# yum -y install dhcp
[root@localhost ~]# vi /etc/dhcp/dhcpd.conf
# create new
# specify domain name
option domain-name "example.com";
# specify name server's hostname or IP address
option domain-name-servers dhcp.example.com;
# default lease time
default-lease-time 600;
# max lease time
max-lease-time 7200;
# this DHCP server to be declared valid
authoritative;
# specify network address and subnet mask
subnet 10.0.0.0 netmask 255.255.255.0 {
    # specify the range of lease IP address
    range dynamic-bootp 10.0.0.200 10.0.0.254;
    # specify broadcast address
    option broadcast-address 10.0.0.255;
    # specify default gateway
    option routers 10.0.0.1;
    optiondomain-name-servers 192.168.188.11, 192.168.188.12
}
[root@localhost ~]# systemctl start dhcpd 
[root@localhost ~]# systemctl enable dhcpd 
ln -s '/usr/lib/systemd/system/dhcpd.service' '/etc/systemd/system/multi-user.target.wants/dhcpd.service'

Dnsmasq is very useful in libvirt/qemu/docker based environment. To bind 192.168.200.0/24 to interface br0(192.168.200.22-bridge2eth1):

1
2
#!/bin/bash
/usr/sbin/dnsmasq --strict-order --pid-file=/var/run/libvirt/network/br0--conf-file= --except-interface lo --bind-interfaces br0 --listen-address 192.168.200.22 --dhcp-range 192.168.200.2,192.168.200.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/br0.leases --dhcp-lease-max=253 --dhcp-no-override --dhcp-hostsfile=/var/lib/libvirt/dnsmasq/br0.hostsfile --addn-hosts=/var/lib/libvirt/dnsmasq/br0.addnhosts

"《KVM私有云架构设计与实践》目录" "2015-09-18"

书籍目前已出版,购买链接在下文提供,也可直接联系作者购买,微信lofyer_org。

当当 亚马逊 京东


"OpenStack自动化部署与高可用虚拟控制节点设计" "2017-05-05"

为了大家的部署方便与维护着想,我决定把这俩东西再完善一下然后再把实现源码放出来,虽不能做到产品级但是也能对付很多场景了。

自动部署(类似mirantis但更友好) 0. livecd+独立配置分区 1. 环境配置 2. PXE安装 3. 自动化脚本

Hosted-crotroller(控制节点虚拟化) 1. etcd 2. 健康检查与评分 3. 脑裂防护 4. 使用Paxos写事务机制,如果时间差机制不工作的话。

下面给大家简单解释一Paxos和时间差机制的原理(解释的不够到位,有兴趣的看实现吧):

  1. Paxos,原文可参考https://www.quora.com/Distributed-Systems-What-is-a-simple-explanation-of-the-Paxos-algorithm

此场景实用于一写多。

两人结婚,牧师问他俩,“你们愿意吗?”,“我愿意”,“我愿意”。 那么OK,就结了。

这时候假如有一对姐妹同时喜欢一个男的,牧师问他仨,“你们愿意吗”,“我愿意”,“我愿意”,“我愿意”。 如果这样,那么男的就能娶俩,如果任何一个没回答,那抱歉了一个都娶不成。

在实现时,主节点问下面节点,我要给你们都写个东西,你们都准备好没?“好了”,“好了”,“好了”,“没好”。 75%(>50%)(Quorum)的人都准备好了,那么我就写了,没准备好的那个我记住你了。

  1. Paxos时间差机制(sanlock变种) 这个机制我设计成只针对这个场景的,要求时间同步至数秒级别(这要求已经极低了。。),多写一,也可以扩展。

控制节点要你们去存储上填个表格,你们现在都能写,对吧?“对”,“对”,“对”,“不能”。。 OK,不能的那个我记住你了,其他人都来写吧。

控制节点看看表,再看看这个地方之前记的时间,嗯,过去2分钟了,写一下,我在A身上跑着; 然后A看看表,再看看这个地方之前记的时间,嗯,还没过去2分钟,不写,之前体检合格,控制节点在我身上; 然后B看看表,再看看这个地方之前记的时间,嗯,过去2分钟了,签个到留个时间,体检优秀; 然后C看看表,再看看这个地方之前记的时间,嗯,过去2分钟了,签个到留个时间,体检合格;

然后控制节点每隔1分钟看一下,嗯,你们都很健康。 又过了3分钟,哎?那个C你没写啊,去跟D站一起。

忽然,大家在填表的时候发现,控制节点都过了3分钟没写了,B就启动一下控制节点。 A发现自己与外界失去联系,就把自己身上都控制节点关了,等待救援。

  1. 基于etcd的时间差机制 上述的简化版,填的表格更少,时间差一致 大家一起写etcd的那个地方。“好”,“好”,“好”。 哎?C你没写,再见。 哎?控制节点呢?来A你检查一下启一下控制节点,B现在状态不佳。 由于控制节点之前在你A上,它现在不行了,扣A个10分,现在分没B高啊。 A你清理下现场让B去启动吧。

试图设计一个任何场景都能应对的功能。

title: "Openstack 配置及其API的简单使用" date: 2013-07-18 categories: - "cloud-infra"


安装除Neutron(Quantum)以外所有东西

在开始之前,假设有三个NIC

禁用selinux,并修改网络配置/etc/sysconfig/network-scripts/ifcfg-ethX

Internal

DEVICE=eth0 TYPE=Ethernet BOOTPROTO=static IPADDR=192.168.1.11 NETMASK=255.255.255.0 GATEWAY=192.168.1.1 DNS1=192.168.1.1 DEFROUTE=yes ONBOOT=yes

External

DEVICE=eth1 TYPE=Ethernet BOOTPROTO=static IPADDR=192.168.1.12 NETMASK=255.255.255.0 DEFROUTE=yes ONBOOT=yes

Public Bridge

DEVICE=eth2 TYPE=Ethernet BOOTPROTO=static IPADDR=192.168.1.13 NETMASK=255.255.255.0 DEFROUTE=yes ONBOOT=yes

于Feodra18中安装Openstack,可参考文档,按照这个来的话对Openstack的设计理念将会有所了解 Getting started with OpenStack on Fedora 18 或者运行

/安装完成后管理员界面为http://localhost/dashboard用户名admin,密码secrete/

openstack-demo-install

以上两种方法把除了Neutron(quantum)其他的服务都配置好了,关于quantum的配置见文末

添加虚拟机实例

可参考文档 Getting started with OpenStack on Fedora 18 在终端操作之前需要设置环境变量,将它们两者之一写到一个文件,然后source一下

export OS_USERNAME=admin export OS_PASSWORD=secrete export OS_TENANT_NAME=admin export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/

export ADMIN_TOKEN=\((openssl rand -hex 10) export SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0/ export SERVICE_TOKEN=\)ADMIN_TOKEN

source FileA

这个libvirt支持kvm以及xen虚拟化,所以支持的镜像也比较多,这里下载一个已经安装好的f17的qcow2镜像

axel -n 5 http://berrange.fedorapeople.org//blog/images/2012-11-15/f17-x86_64-openstack-sda.qcow2

/注册镜像/

glance add name=f17-jeos is_public=true disk_format=qcow2 container_format=bare < f17-x86_64-openstack-sda.qcow2

然后以demo用户进入dashboard,会看到添加进来的image,根据image启动实例,打开vnc界面(非localhost访问要改ip)。

安装操蛋的Neutron(quantum)服务

这东西刚改名,nova自带的有network服务,可这个可以虚拟L2 L3 switch,那就比较有意思了

/注册quantum/

keystone service-create --name quantum --type network --description 'OpenStack Networking Service'

keystone endpoint-create --region myregion --service-id 26a55b340e254ad5bb78c0b14391e153 --publicurl "http://192.168.1.11:9696/" --adminurl "http://192.168.1.11:9696/" --internalurl "http://192.168.1.11:9696/"

/(可选)添加quantum服务用户/ get_id.sh:

$ function get_id () { echo "$@" | awk '/ id / { print $4 }' }

ADMIN_ROLE=$(get_id keystone role-create --name=admin)

QUANTUM_USER=$(get_id keystone user-create --name=quantum --pass="servicepass" --email=demo@example.com --tenant-id service)

keystone user-role-add --user_id $QUANTUM_USER --role_id $ADMIN_ROLE --tenant_id service

/服务插件二选一,linuxbrideg和openvswitch,这里使用后者,先安装插件/

yum install openstack-quantum-openvswitch

/启动并添加服务/

quantum-server-setup --plugin openvswitch

quantum-node-setup --plugin openvswitch

service quantum-server start

chkconfig quantum-server on

service openvswitch start

chkconfig openvswitch on

ovs-vsctl add-br br-int

chkconfig quantum-openvswitch-agent on

service quantum-openvswitch-agent start

quantum-dhcp-setup --plugin openvswitch

chkconfig quantum-dhcp-agent on

service quantum-dhcp-agent start

ovs-vsctl add-br br-ex

ovs-vsctl add-port br-ex eth1

quantum-l3-setup --plugin openvswitch

chkconfig quantum-l3-agent on

service quantum-l3-agent start

chkconfig quantum-metadata-agent on

service quantum-metadata-agent start

修改并配置网络 /etc/sysconf/network-scripts/ifcfg-eth1,/etc/sysconf/network-scripts/ifcfg-br-ex

External

DEVICE=eth1 TYPE=Ethernet BOOTPROTO=none NM_CONTROLLED=no BRIDGE=br-ex ONBOOT=yes

Public Bridge

DEVICE=br-ex TYPE=Bridge BOOTPROTO=static IPADDR=192.168.2.11 NETMASK=255.255.255.0 NM_CONTROLLED=no ONBOOT=yes

ip addr del 10.0.0.9/24 dev eth1

ip addr add 10.0.0.9/24 dev br-ex

这里类似VPN的重写HEADER

echo 1 > /proc/sys/net/ipv4/conf/all/forwarding sysctl -w net.ipv4.ip_forward=1 iptables -A FORWARD -i eth0 -o br-ex -s 10.10.10.0/24 -m conntrack --ctstate NEW -j ACCEPT iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT iptables -A POSTROUTING -s 10.10.10.0/24 -t nat -j MASQUERADE

最后重启机器即可

通过curl使用openstack api

参考这篇文章 注意区分token和固定id

获取可用镜像

curl -s http://10.199.0.250:8774/v2/51ad87714b86442d9a74537d6f890060/images -X GET -H "X-Auth-Project-Id: admin" -H "Accept: application/json" -H "X-Auth-Token: 6083193874684f38865b086a8a5f4b7b" | python -mjson.tool

创建虚拟机

curl -i http://10.199.0.250:8774/v2/51ad87714b86442d9a74537d6f890060/servers -X POST -H "X-Auth-Project-Id: admin" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: 6083193874684f38865b086a8a5f4b7b" -d '{"server": {"name": "instance1", "imageRef": "992f5732-af50-40fc-987f-25951cbce943", "key_name": "damion-flybook", "flavorRef": "3", "max_count": 1, "min_count": 1}}'


title: "A design of user frontend for kinds of Cloud with accounting." date: 2015-05-06 categories: - "cloud-infra" tags: - "Cloud Computing" - "TBD"


I like Linode.com very much, so I'd like to make a frontend with something like that in functions.

Here's the draft.

Untitled Page

更新一下,两年前的架构设计,终于在2017年的客户那用上了,而且上了人工干预的运行前调度器,哇哈哈。

title: "Cloud Init how to" date: 2015-03-10 categories: - "cloud-infra"


OS: CentOS 6

yum install -y cloud-init


title: "How to solve the 'boot storm' problem: BCACHE" date: 2013-05-27 categories: - "cloud-infra" - "linux-admin"


Some solutions Way 1. Add more cpus.(ABANDONED) Way 2. Add a SSD as "boot cache" Way 3. Sort the boot process

POC: Way 2: Sometimes we need cache the boot section of the OS into a SSD, since no SSD on hand, let's try to use a block device made in /dev/shm

Way 3: Considering that the parallelization of the "boot action", we have to predict the action in the near future

Experiment: Using BCACHE now...or flashcache

title: "A light-weighted VM webadmin" date: 2013-06-18 categories: - "cloud-infra" - "linux-admin"


The infrastructure of it: Management: Based on django, communicate with Agent Agent: A daemon, used for communicating with Management and managing the VMs

TO MAKE GREAT USE OF KVM. https://github.com/lofyer/webadmin

title: "MyIDC--Everything's traceable" date: 2013-12-11 categories: - "cloud-infra"


相关手册参考MyIDC_Book

细节索引: gluster hadoop(with ARM as an option) drbd/heartbeat foreman/puppet nagios ovirt openstack owncloud asterisk

配置细节参考: MyIDC Source

Infrastructure:

Board: x86小板 SBSA小板 Cabinet: 亚克力自制 网络: 电信拨号接入外网 SDN 资源分配: 高可用监视与控制服务(os1-2) 高可用存储(os3-n) 并行计算(os3-n) 高可用虚拟机服务(os1/2+os3-n) 独立外设接入设备 安全: 证书认证(在量子计算机面向大众之前这个还是很靠谱的) 应用商店: 社会工程学测试 leapmotion控制插件 语音控制插件 手机控制插件


title: "openvswitch howto" date: 2013-07-25 categories: - "linux-admin"


Here, I use kernel-3.4.58.

emerge =net-misc/openvswitch-1.11.0


title: "硬盘、RAID组与Ceph分布式存储IOPS计算公式" date: 2017-06-04 categories: - "linux-admin" tags: - "Storage" - "Performance"


防怼说明:本文用于速算,想把难以量化的环境变量以及各种组合拿来说事儿的请自行Google,或者问厂商和IDC要报告去。

机械硬盘

1
2
3
7200硬盘IOPS = 1000/(3 + 1000*(7200/60)/2) = 140
10k硬盘IOPS = 1000/(3 + 60000/10000/2) = 167
15k硬盘IOPS = 1000/(3 + 60000/15000/2) = 200

其中3为寻道延迟,7200/10k/15k为转速(rpm),1000*(7200/60)/2为旋转延迟(旋转延迟一般用转一圈所需时间的1/2表示),结果为理论峰值,实际还会有系统延迟导致测得IOPS一般低于此值。

RAID组 由于RAID组需要校验以提供恢复功能,所以会存在一定写惩罚,这个系数如下: RAID0: 1 RAID1: 2 RAID5: 4 RAID6: 6 RAID1-0: 2

所以RAID组IOPS = 硬盘写IOPS硬盘数量写操作百分比/写惩罚系数 + 硬盘读IOPS硬盘数量读操作百分比。

以4块IOPS为180的SAS硬盘组RAID 6然后百分百随机写操作为例:

IOPS = 180*4/6 = 120

Ceph的IOPS经验公式 由于Ceph存储结构不同于物理硬件,所以影响其IOPS的因素主要有网络、副本数量、日志、OSD(硬盘)数量、OSD服务器数量、OSD IOPS等,这里给出一个来自Mirantis的经验公式:

IOPS = 硬盘IOPS * 硬盘数量 * 0.88 / 副本数量, 其中0.88为4-8k随机读操作占比(88%)。

关于Ceph的IOPS计算仅供参考,计算结果可能会跟物理环境实测有较大偏差。