Python





```python
file_path="./log_record/"
if not os.path.exists(file_path):
    os.makedirs(file_path)
file_name=file_path+'log.log'
if not os.path.exists(file_name):
    os.system(r"touch {}".format(file_name))  # 调用系统命令行来创建文件
```

python 检测文件目录是否存在 不存在就创建 检测文件是否存在 不存在就创建 linux

Python





```python
import datetime
print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
```

<table>

 <thead>

  <tr>

   <td><strong>Directive</strong></td>

   <td><strong>Meaning</strong></td>

   <td><strong>Example</strong></td>

  </tr>

 </thead>

 <tbody>

  <tr>

   <td><code>%a</code></td>

   <td>Abbreviated weekday name.</td>

   <td>Sun, Mon, ...</td>

  </tr>

  <tr>

   <td><code>%A</code></td>

   <td>Full weekday name.</td>

   <td>Sunday, Monday, ...</td>

  </tr>

  <tr>

   <td><code>%w</code></td>

   <td>Weekday as a decimal number.</td>

   <td>0, 1, ..., 6</td>

  </tr>

  <tr>

   <td><code>%d</code></td>

   <td>Day of the month as a zero-padded decimal.</td>

   <td>01, 02, ..., 31</td>

  </tr>

  <tr>

   <td><code>%-d</code></td>

   <td>Day of the month as a decimal number.</td>

   <td>1, 2, ..., 30</td>

  </tr>

  <tr>

   <td><code>%b</code></td>

   <td>Abbreviated month name.</td>

   <td>Jan, Feb, ..., Dec</td>

  </tr>

  <tr>

   <td><code>%B</code></td>

   <td>Full month name.</td>

   <td>January, February, ...</td>

  </tr>

  <tr>

   <td><code>%m</code></td>

   <td>Month as a zero-padded decimal number.</td>

   <td>01, 02, ..., 12</td>

  </tr>

  <tr>

   <td><code>%-m</code></td>

   <td>Month as a decimal number.</td>

   <td>1, 2, ..., 12</td>

  </tr>

  <tr>

   <td><code>%y</code></td>

   <td>Year without century as a zero-padded decimal number.</td>

   <td>00, 01, ..., 99</td>

  </tr>

  <tr>

   <td><code>%-y</code></td>

   <td>Year without century as a decimal number.</td>

   <td>0, 1, ..., 99</td>

  </tr>

  <tr>

   <td><code>%Y</code></td>

   <td>Year with century as a decimal number.</td>

   <td>2013, 2019 etc.</td>

  </tr>

  <tr>

   <td><code>%H</code></td>

   <td>Hour (24-hour clock) as a zero-padded decimal number.</td>

   <td>00, 01, ..., 23</td>

  </tr>

  <tr>

   <td><code>%-H</code></td>

   <td>Hour (24-hour clock) as a decimal number.</td>

   <td>0, 1, ..., 23</td>

  </tr>

  <tr>

   <td><code>%I</code></td>

   <td>Hour (12-hour clock) as a zero-padded decimal number.</td>

   <td>01, 02, ..., 12</td>

  </tr>

  <tr>

   <td><code>%-I</code></td>

   <td>Hour (12-hour clock) as a decimal number.</td>

   <td>1, 2, ... 12</td>

  </tr>

  <tr>

   <td><code>%p</code></td>

   <td>Locale&rsquo;s AM or PM.</td>

   <td>AM, PM</td>

  </tr>

  <tr>

   <td><code>%M</code></td>

   <td>Minute as a zero-padded decimal number.</td>

   <td>00, 01, ..., 59</td>

  </tr>

  <tr>

   <td><code>%-M</code></td>

   <td>Minute as a decimal number.</td>

   <td>0, 1, ..., 59</td>

  </tr>

  <tr>

   <td><code>%S</code></td>

   <td>Second as a zero-padded decimal number.</td>

   <td>00, 01, ..., 59</td>

  </tr>

  <tr>

   <td><code>%-S</code></td>

   <td>Second as a decimal number.</td>

   <td>0, 1, ..., 59</td>

  </tr>

  <tr>

   <td><code>%f</code></td>

   <td>Microsecond as a decimal number, zero-padded on the left.</td>

   <td>000000 - 999999</td>

  </tr>

  <tr>

   <td><code>%z</code></td>

   <td>UTC offset in the form +HHMM or -HHMM.</td>

   <td>&nbsp;</td>

  </tr>

  <tr>

   <td><code>%Z</code></td>

   <td>Time zone name.</td>

   <td>&nbsp;</td>

  </tr>

  <tr>

   <td><code>%j</code></td>

   <td>Day of the year as a zero-padded decimal number.</td>

   <td>001, 002, ..., 366</td>

  </tr>

  <tr>

   <td><code>%-j</code></td>

   <td>Day of the year as a decimal number.</td>

   <td>1, 2, ..., 366</td>

  </tr>

  <tr>

   <td><code>%U</code></td>

   <td>Week number of the year (Sunday as the first day of the week). All days in a new year preceding the first Sunday are considered to be in week 0.</td>

   <td>00, 01, ..., 53</td>

  </tr>

  <tr>

   <td><code>%W</code></td>

   <td>Week number of the year (Monday as the first day of the week). All days in a new year preceding the first Monday are considered to be in week 0.</td>

   <td>00, 01, ..., 53</td>

  </tr>

  <tr>

   <td><code>%c</code></td>

   <td>Locale&rsquo;s appropriate date and time representation.</td>

   <td>Mon Sep 30 07:06:05 2013</td>

  </tr>

  <tr>

   <td><code>%x</code></td>

   <td>Locale&rsquo;s appropriate date representation.</td>

   <td>09/30/13</td>

  </tr>

  <tr>

   <td><code>%X</code></td>

   <td>Locale&rsquo;s appropriate time representation.</td>

   <td>07:06:05</td>

  </tr>

  <tr>

   <td><code>%%</code></td>

   <td>A literal &#39;%&#39; character.</td>

   <td>%</td>

  </tr>

 </tbody>

</table>

python datetime to string using strftime()

Python





```python
if __name__ == '__main__':
    result_generator = (dict(zip([5,6], x)) for x in [(1,2),(1,2),(1,2)])  # 小括号 生成器
    print(next(result_generator))
    print(next(result_generator))
    # print(next(result_generator))
    for i in range(4):
        try:
            try:
                print(next(result_generator))
            except:
                print("空的")
            else:
                continue
            print(i)
        except:
            print("dsad")
```

python 生成器 是否为空 判断 generator

[https://docs.python.org/3/library/pickle.html](https://docs.python.org/3/library/pickle.html)





pickle协议和JSON（JavaScript对象表示法）之间存在根本差异 ：

JSON是一种文本序列化格式（虽然大多数时间然后将其编码为，但它输出unicode文本utf-8），而pickle是一种二进制序列化格式；

JSON是人类可读的，而pickle则不是；

JSON是可互操作的，并且在Python生态系统之外被广泛使用，而pickle是特定于Python的；

默认情况下，JSON只能表示Python内置类型的子集，而不能表示自定义类。pickle可以表示大量的Python类型（通过巧妙地使用Python的自省功能，其中的许多会自动实现；可以通过实现特定的对象API解决复杂的情况）；

与pickle不同，反序列化不受信任的JSON本身并不创建任意代码执行漏洞。


总结：

1 json好看，但内部类型少。可跨平台。

2 pickle不好看，但是叼，啥都能存，还能复原。

3 建议：不整活就pickle就行了，好用，兼容好。

4 怎么用：

存成一个文件：

```python
import pickle
with open('temp_debug_t_all.txt','wb') as f:
    pickle.dump(t_all, f)
```

读取出来

```python
import pickle
with open('temp_debug_t_all.txt', 'rb') as f:
    t_all = pickle.load(f)
```

python 保存变量 pickle pickle — Python object serialization

实用写法traceback：





```python
import traceback
try:
   a=1/0
except:
    traceback.print_exc()
finally:
    print("有无异常都执行一次")
```

捕获指定异常：
```python
try:
   a=1/0
except ZeroDivisionError:
    print("捕获指定的异常")
finally:
    print("有无异常都执行一次")
```

瞎写：
```python
import traceback
try:
   a=1/0
except:
    print("有异常 但也不知道异常是啥")
else:
    print("没有异常才会执行")
```

```python
BaseException  # 所有异常的基类
 +-- SystemExit  # 解释器请求退出
 +-- KeyboardInterrupt  # 用户中断执行(通常是输入^C)
 +-- GeneratorExit  # 生成器(generator)发生异常来通知退出
 +-- Exception  # 常规异常的基类
      +-- StopIteration  # 迭代器没有更多的值
      +-- StopAsyncIteration  # 必须通过异步迭代器对象的__anext__()方法引发以停止迭代
      +-- ArithmeticError  # 各种算术错误引发的内置异常的基类
      |    +-- FloatingPointError  # 浮点计算错误
      |    +-- OverflowError  # 数值运算结果太大无法表示
      |    +-- ZeroDivisionError  # 除(或取模)零 (所有数据类型)
      +-- AssertionError  # 当assert语句失败时引发
      +-- AttributeError  # 属性引用或赋值失败
      +-- BufferError  # 无法执行与缓冲区相关的操作时引发
      +-- EOFError  # 当input()函数在没有读取任何数据的情况下达到文件结束条件(EOF)时引发
      +-- ImportError  # 导入模块/对象失败
      |    +-- ModuleNotFoundError  # 无法找到模块或在在sys.modules中找到None
      +-- LookupError  # 映射或序列上使用的键或索引无效时引发的异常的基类
      |    +-- IndexError  # 序列中没有此索引(index)
      |    +-- KeyError  # 映射中没有这个键
      +-- MemoryError  # 内存溢出错误(对于Python 解释器不是致命的)
      +-- NameError  # 未声明/初始化对象 (没有属性)
      |    +-- UnboundLocalError  # 访问未初始化的本地变量
      +-- OSError  # 操作系统错误，EnvironmentError，IOError，WindowsError，socket.error，select.error和mmap.error已合并到OSError中，构造函数可能返回子类
      |    +-- BlockingIOError  # 操作将阻塞对象(e.g. socket)设置为非阻塞操作
      |    +-- ChildProcessError  # 在子进程上的操作失败
      |    +-- ConnectionError  # 与连接相关的异常的基类
      |    |    +-- BrokenPipeError  # 另一端关闭时尝试写入管道或试图在已关闭写入的套接字上写入
      |    |    +-- ConnectionAbortedError  # 连接尝试被对等方中止
      |    |    +-- ConnectionRefusedError  # 连接尝试被对等方拒绝
      |    |    +-- ConnectionResetError    # 连接由对等方重置
      |    +-- FileExistsError  # 创建已存在的文件或目录
      |    +-- FileNotFoundError  # 请求不存在的文件或目录
      |    +-- InterruptedError  # 系统调用被输入信号中断
      |    +-- IsADirectoryError  # 在目录上请求文件操作(例如 os.remove())
      |    +-- NotADirectoryError  # 在不是目录的事物上请求目录操作(例如 os.listdir())
      |    +-- PermissionError  # 尝试在没有足够访问权限的情况下运行操作
      |    +-- ProcessLookupError  # 给定进程不存在
      |    +-- TimeoutError  # 系统函数在系统级别超时
      +-- ReferenceError  # weakref.proxy()函数创建的弱引用试图访问已经垃圾回收了的对象
      +-- RuntimeError  # 在检测到不属于任何其他类别的错误时触发
      |    +-- NotImplementedError  # 在用户定义的基类中，抽象方法要求派生类重写该方法或者正在开发的类指示仍然需要添加实际实现
      |    +-- RecursionError  # 解释器检测到超出最大递归深度
      +-- SyntaxError  # Python 语法错误
      |    +-- IndentationError  # 缩进错误
      |         +-- TabError  # Tab和空格混用
      +-- SystemError  # 解释器发现内部错误
      +-- TypeError  # 操作或函数应用于不适当类型的对象
      +-- ValueError  # 操作或函数接收到具有正确类型但值不合适的参数
      |    +-- UnicodeError  # 发生与Unicode相关的编码或解码错误
      |         +-- UnicodeDecodeError  # Unicode解码错误
      |         +-- UnicodeEncodeError  # Unicode编码错误
      |         +-- UnicodeTranslateError  # Unicode转码错误
      +-- Warning  # 警告的基类
           +-- DeprecationWarning  # 有关已弃用功能的警告的基类
           +-- PendingDeprecationWarning  # 有关不推荐使用功能的警告的基类
           +-- RuntimeWarning  # 有关可疑的运行时行为的警告的基类
           +-- SyntaxWarning  # 关于可疑语法警告的基类
           +-- UserWarning  # 用户代码生成警告的基类
           +-- FutureWarning  # 有关已弃用功能的警告的基类
           +-- ImportWarning  # 关于模块导入时可能出错的警告的基类
           +-- UnicodeWarning  # 与Unicode相关的警告的基类
           +-- BytesWarning  # 与bytes和bytearray相关的警告的基类
           +-- ResourceWarning  # 与资源使用相关的警告的基类。被默认警告过滤器忽略。
```

参考：https://www.cnblogs.com/beile/p/10789333.html

python traceback python捕获异常及方法总结

[https://docs.python.org/zh-cn/3/library/asyncio-task.html#timeouts](https://docs.python.org/zh-cn/3/library/asyncio-task.html#timeouts)





适合IO密集型任务，可以当成轻线程去理解（不需要线程的那种上下文切换）。

```python
import asyncio
import datetime
async def display_date(num, loop):  # 注意这一行的写法
    '''
    这个函数执行一次需要五秒
    '''
    end_time = loop.time() + 5.0
    while True:
        print("我的任务编号: {} Time: {}".format(num, datetime.datetime.now()))
        if (loop.time() + 1.0) >= end_time:
            break
        await asyncio.sleep(1)  # 阻塞直到协程sleep(2)返回结果
if __name__ == '__main__':
    loop = asyncio.get_event_loop()  # 获取一个event_loop
    tasks = [display_date(1, loop), display_date(2, loop)]
    loop.run_until_complete(asyncio.gather(*tasks))  # "阻塞"直到所有的tasks完成
    loop.close()
```

python 协程 async await

Python





```python
from collections import defaultdict
def with_running_count(values):
    counts = defaultdict(int)
    for value in values:
        counts[value] += 1
        yield value, counts[value]
print(*with_running_count('abaab'))
```

yield 生成器 生成 *

[https://docs.python.org/3/library/multiprocessing.html](https://docs.python.org/3/library/multiprocessing.html)





# 1 背景

python由于[Global Interpreter Lock](https://docs.python.org/3/glossary.html#term-global-interpreter-lock) 的原因，多线程并不是我们想的那种CPU调度的多线程，在python中开了多线程其实也是一个单进程。

针对CPU密集型（CPU-bound）的任务，比如计算气象数据，多线程和多进程调用多个CPU核心并行，能有效减少任务计算时间，提高实时性，但python的多线程是不ok的。

针对IO密集型（I/O bound）的任务，比如爬虫，多线程和多进程也是OK的，python的多线程显然也是ok的。

进程会占用更多资源，比如主进程开一个子进程，子进程会把主进程的变量啥的复制一份，这也导致进程之间通信和线程不一样。多线程的时候，给一个全局列表进去还能线程安全；多进程的时候，子进程只会把主进程的全局列表复制一份带入，修改的也是修改子进程的。所以需要类似信号、管道、消息队列、共享内存的东西来进行，python multiprocessing有消息队列、管道、共享内存，优点是快，缺点是不太灵活（存储类型和运行机制），python multiprocessing提供了一个更高级的封装类`Manager`来专门处理这个事情，`Manager`自己就是一个服务进程，有点就是灵活，缺点就是慢一些。


# 2 进程池

进程池肯定是很爽的，使用简单，效果显著。

windows需要multiprocessing.freeze_support()

Linux不需要。


```python
import multiprocessing
import time
import os
def func1(inputdata, res):
    res.append([sum(inputdata)])
    print("res此时的结果： ", res)
    print("此时的时间： ", time.time())
    print("进城的PID： ", os.getpid())
if __name__ == '__main__':
    multiprocessing.freeze_support()
    res = multiprocessing.Manager().list()  # 通信
    inputdata = [i for i in range(100)]  # 随便弄点数
    p = multiprocessing.Pool(2)  # 创建一个包含2个进程的进程池
    p.apply_async(func=func1, args=(inputdata[0:10], res,))  # 往池子里加一个异步执行的子进城
    p.apply_async(func=func1, args=(inputdata[10:20], res,))  # 往池子里加一个异步执行的子进程
    p.apply_async(func=func1, args=(inputdata[20:30], res,))  # 往池子里加一个异步执行的子进程
    p.close()  # 等子进程执行完毕后关闭进程池
    p.join()  # 主进程等待
```

运行结果：

```python
res此时的结果：  [[45]]
此时的时间：  1610254954.318595
进城的PID：  15372
res此时的结果：  [[45], [245]]
此时的时间：  1610254954.3205965
进城的PID：  15372
res此时的结果：  [[45], [245], [145]]
此时的时间：  1610254954.321597
进城的PID：  16268
```

multiprocessing — Process-based parallelism python 多进程 用法 进程池

# pymysql封装





只封装了select选择数据和insert插入数据：
```python
# encoding: utf-8
import traceback
import pymysql
import pandas as pd
class easy_pymysql():
    # 初始化
    def __init__(self, host, user, passwd):
        self.host = host
        self.user = user
        self.passwd = passwd
        self.conn = None
        self.cursor = None
    def connect(self):
        self.conn = pymysql.connect(host=self.host,
                                    user=self.user,
                                    password=self.passwd,
                                    # database=self.dbName,
                                    use_unicode=True,
                                    charset='utf8')
        self.cursor = self.conn.cursor()
    def close(self):
        self.cursor.close()
        self.conn.close()
    def select_data(self, sql):
        result = None
        self.connect()
        try:
            self.cursor.execute(sql)
            result = self.cursor.fetchall()
            result = pd.DataFrame(result)
        except:
            traceback.print_exc()
            self.conn.rollback()
        self.close()
        return result
    def insert(self, df: pd.DataFrame, tablename: str):
        df = df.to_dict(orient="split")
        tup_col = ", ".join(df['columns'])
        datas = str([tuple(i) for i in df['data']])[1:-1]
        sql = f"INSERT INTO {tablename} ({tup_col}) VALUES {datas}"
        self.connect()
        try:
            self.cursor.execute(sql)
            self.conn.commit()
        except:
            traceback.print_exc()
            self.conn.rollback()
        self.close()
    def insert_table(self, df: pd.DataFrame, tablename: str):
        if len(df) <= 20000:
            self.insert(df, tablename)
        else:
            n = len(df) // 20000
            for i in range(n):
                df_sub = df[20000 * i:(20000 * (i + 1))]
                self.insert(df_sub, tablename)
            df_sub = df[(20000 * (i + 1)):]
            self.insert(df_sub, tablename)
        print(f"finish transfer mysql {len(df)}")
    def update_table(self, df: pd.DataFrame, tablename: str):
        self.connect()
        try:
            idchname, upchname = list(df.columns)
            print(idchname, upchname)
            id_list = list(df[idchname])
            upch_list = list(df[upchname])
            for id_, upch_ in zip(id_list, upch_list):
                sql = f"UPDATE {tablename} SET {upchname}={upch_} WHERE {idchname}={id_}"
                self.cursor.execute(sql)
            self.conn.commit()
        except:
            traceback.print_exc()
            self.conn.rollback()
        self.close()
```

# clickhouse_driver封装

只封装了select选择数据和insert插入数据：

```python
# encoding: utf-8
from clickhouse_driver.client import Client
import pandas as pd
class easy_clickhouse():
    # 初始化
    # def __init__(self, host, user="", passwd=""):
    def __init__(self, host):
        self.host = host
        self.client = Client(host=self.host)
    def select_data(self, sql):
        result = self.client.execute(sql)
        if len(result) == 0:
            return None
        else:
            result = pd.DataFrame(result)
        return result
    def insert(self, df: pd.DataFrame, tablename: str):
        df = df.to_dict(orient="split")
        tup_col = ", ".join(df['columns'])
        datas = str([tuple(i) for i in df['data']])[1:-1]
        sql = f"INSERT INTO {tablename} ({tup_col}) VALUES {datas}"
        self.client.execute(sql)
    def insert_table(self, df: pd.DataFrame, tablename: str):
        if len(df) <= 20000:
            self.insert(df, tablename)
        else:
            n = len(df) // 20000
            for i in range(n):
                df_sub = df[20000 * i:(20000 * (i + 1))]
                self.insert(df_sub, tablename)
            df_sub = df[(20000 * (i + 1)):]
            self.insert(df_sub, tablename)
        print(f"finish transfer clickhouse {len(df)}")
```

```python
def insert_db_table(datadict_list):
    '''
    datadict_list是字典列表  每个字典是一条记录
    开发环境的
    '''
    host = '10.10.90.11'
    table = "tbs"
    sql = "insert into %s " % (table)
    c = clientclickhouse(host=host)
    sum1 = 0
    for datadict in datadict_list:
        result = datadict  # 字典
        names = ["%s" % x for x in result.keys()]
        names = ",".join(names)
        if sum1 == 0:
            sql = sql + "(%s) values " % (names)
        values = ["'%s'" % x if isinstance(x, str) else "{}".format(x) for x in result.values()]
        values = ",".join(values)
        sql = sql + "(%s)," % (values)
        sum1 = sum1 + 1
    sql_str = sql[0:-1]
    c.execute(sql_str)
```

python pymysql 插入多条数据 字典列表 数据 clickhouse driver 封装 SQL

原文：





[https://blog.csdn.net/dl1456074580/article/details/86538098](https://blog.csdn.net/dl1456074580/article/details/86538098)

```python
def skip_ws(txt, pos):
    while pos < len(txt) and txt[pos].isspace():
        pos += 1
    return pos
def parse_str(txt, pos, allow_ws=False, delimiter=[',', ':', '}', ']']):
    while pos < len(txt):
        if not allow_ws and txt[pos].isspace():
            break
        if txt[pos] in delimiter:
            break
        pos += 1
    return pos
def parse_obj(txt, pos):
    obj = dict()
    while True:
        pos = skip_ws(txt, pos + 1)
        end = parse_str(txt, pos, True, [':'])
        if end >= len(txt):
            raise ValueError("unexpected end when parsing object key")
        key = txt[pos:end].strip()
        pos = skip_ws(txt, end + 1)
        if pos >= len(txt):
            raise ValueError("unexpected end when parsing object value")
        if txt[pos] == '[':
            value, pos = parse_array(txt, pos)
        elif txt[pos] == '{':
            value, pos = parse_obj(txt, pos)
        else:
            end = parse_str(txt, pos, True, [',', '}'])
            if end >= len(txt):
                raise ValueError("unexpected end when parsing object value")
            value = txt[pos:end].strip()
            pos = end
        obj[key] = value
        pos = skip_ws(txt, pos)
        if pos >= len(txt):
            raise ValueError("unexpected end when object value finish")
        if txt[pos] == '}':
            return obj, pos + 1
def parse_array(txt, pos):
    array = list()
    while True:
        pos = skip_ws(txt, pos + 1)
        if pos >= len(txt):
            raise ValueError("unexpected end when parsing array item")
        if txt[pos] == '[':
            value, pos = parse_array(txt, pos)
        elif txt[pos] == '{':
            value, pos = parse_obj(txt, pos)
        else:
            end = parse_str(txt, pos, True, [',', ']'])
            if end >= len(txt):
                raise ValueError("unexpected end when parsing array item")
            value = txt[pos:end].strip()
            pos = end
        array.append(value)
        pos = skip_ws(txt, pos)
        if pos >= len(txt):
            raise ValueError("unexpected end when array item finish")
        if txt[pos] == ']':
            return array, pos + 1
def parse(txt):
    if txt.startswith('json'):
        pos = txt.find(':')
        if pos != -1:
            pos = skip_ws(txt, pos + 1)
            if txt[pos] == '{':
                obj, pos = parse_obj(txt, pos)
                return obj
            elif txt[pos] == '[':
                array, pos = parse_array(txt, pos)
                return array
    raise ValueError("format error when parsing root")
if __name__ == '__main__':
    import json
    txt = r'json:{name:张三,age:22,book:[{bookname:语文,booksize:200,soce:100},{wkke:123}]}'
    resdict = parse(txt)
    print(resdict)
    print(type(resdict))
    restxt=json.dumps(resdict,ensure_ascii=False)
    print(restxt)
    print(type(restxt))
    resd=json.loads(restxt)
    print(resd)
    print(type(resd))
```