Commit Graph

1689 Commits

Author SHA1 Message Date
CJ Cobb
64d0c5d050 async populate caches 2021-05-03 21:18:38 +00:00
CJ Cobb
971437f456 async write to keys and books with recovery and warnings 2021-05-03 21:18:38 +00:00
CJ Cobb
4e58c76eac Fix crashing bug related to account_tx writes 2021-05-03 21:18:36 +00:00
Nathan Nichols
ee763ec1ea merge master 2021-05-03 12:13:50 -05:00
CJ Cobb
c0612e740e Many changes around ledger_data and book_offers
* populate caches on startup
* implement new algorithms for Postgres
* use one shift interval instead of two
* block etl at flag ledger until writes are finished
* write one flag ledger ahead
* abandon ledger_diff component of ledger_data
2021-04-29 21:30:05 +00:00
CJ Cobb
f1ff81ddc5 postgres support for new books and keys tables 2021-04-29 14:51:03 +00:00
CJ Cobb
1f2d670e2b Merge branch 'perf_test_clean' into master_new 2021-04-28 20:33:15 +00:00
CJ Cobb
b734884f20 various improvements around testing 2021-04-28 20:27:52 +00:00
CJ Cobb
8c6fe09590 compilation fix after merge 2021-04-28 20:16:47 +00:00
CJ Cobb
e747c61873 Merge branch 'ledger_data_indexer' into master_new 2021-04-28 20:12:28 +00:00
CJ Cobb
20184527af Merge branch 'postgres_partitioning' into master_new 2021-04-28 19:55:25 +00:00
CJ Cobb
4349668000 Merge branch 'parallel_etl' into master_new 2021-04-28 19:55:08 +00:00
CJ Cobb
5ebc515978 support for read replicas with postgres 2021-04-28 19:02:44 +00:00
CJ Cobb
0cfce33724 drop hash index on objects.ledger_sequence. was making initial ingestion slow 2021-04-28 18:44:21 +00:00
CJ Cobb
40d136b3fc expand primary key on account_transactions to support index only scans. fix account_tx stored procedure 2021-04-28 17:04:00 +00:00
CJ Cobb
3aed708262 make threadpool a class object 2021-04-27 16:19:45 +00:00
Nathan Nichols
3c8ecc01f1 impliment cursor for book_offers 2021-04-26 15:35:29 -05:00
CJ Cobb
3be12dd4b4 time account_tx_full 2021-04-26 14:35:34 +00:00
Nathan Nichols
0bcf3a4601 Order book offers by quality 2021-04-21 23:27:14 -05:00
CJ Cobb
1b7d672427 fix cursor issue with account_tx stored procedure 2021-04-20 17:20:35 +00:00
CJ Cobb
777dbb5826 account_tx stored procedure 2021-04-20 16:09:22 +00:00
CJ Cobb
e833bfb38c simulate async fetch batch transactions 2021-04-16 18:53:05 +00:00
CJ Cobb
5364830224 fix postgres account_tx performance issue 2021-04-16 16:02:44 +00:00
CJ Cobb
d9a8ff5399 index during ETL. not tested 2021-04-15 14:10:28 +00:00
CJ Cobb
ed4ce93481 remove mention of keys table 2021-04-15 13:48:47 +00:00
CJ Cobb
d5f21ab295 fix indexes 2021-04-14 21:33:11 +00:00
CJ Cobb
23ba45ea5d remove main tx hash index 2021-04-14 14:00:28 +00:00
CJ Cobb
dde44aa3e0 table partitioning 2021-04-12 21:14:29 +00:00
CJ Cobb
990e618fe8 add accum parameter 2021-04-12 15:27:16 +00:00
CJ Cobb
67f98e8ac9 fix memory leak 2021-04-08 20:09:21 +00:00
CJ Cobb
d773fcfdb7 add finish sequence 2021-04-07 13:43:24 +00:00
CJ Cobb
5f9e5d03f4 speed up indexer 2021-04-05 15:22:07 +00:00
CJ Cobb
4f834fc25f fix memory leak 2021-04-01 23:43:54 +00:00
CJ Cobb
df8b6f15d9 make cassandra driver thread count configurable 2021-04-01 23:43:42 +00:00
CJ Cobb
0d11898730 indexer picks up from where it left off 2021-04-01 23:43:11 +00:00
CJ Cobb
db37c05b7b new book offers algorithm, seems to work 2021-03-31 19:22:53 +00:00
CJ Cobb
7f39378268 test script updates 2021-03-31 15:57:02 +00:00
Ubuntu
d27b53e4f7 actually write to keys table 2021-03-30 19:36:55 +00:00
Ubuntu
ef555b00a7 indexer values look good. Nothing is being written yet 2021-03-30 18:35:40 +00:00
CJ Cobb
d1f47b490a iterate through diffs. don't write anything 2021-03-30 11:13:51 -04:00
Ubuntu
2eca3cac96 enhance test.py 2021-03-26 19:41:52 +00:00
Ubuntu
8aaa365e46 bypass validated and hasLedger checks 2021-03-26 17:14:05 +00:00
CJ Cobb
9e1bc896e7 fix memory leak 2021-03-26 13:05:48 -04:00
CJ Cobb
58509093ac make cassandra driver thread count configurable 2021-03-26 08:42:23 -04:00
CJ Cobb
d2da6051d1 reduce etl queue size based on number of extractor threads 2021-03-25 21:58:33 -04:00
CJ Cobb
168283f0aa the best ledger_data implementation yet 2021-03-25 21:57:04 -04:00
CJ Cobb
2af6d72d7e reduce etl queue size based on number of extractor threads 2021-03-25 21:57:04 -04:00
CJ Cobb
d72867ec64 switch deserialization order 2021-03-24 21:00:49 -04:00
CJ Cobb
29424fd863 remove sparsity algorithm in cassandra ledger page 2021-03-24 20:44:39 -04:00
Ubuntu
a77d13449d update test.py and metrics.py 2021-03-25 00:42:44 +00:00