CJ Cobb
64d0c5d050
async populate caches
2021-05-03 21:18:38 +00:00
CJ Cobb
971437f456
async write to keys and books with recovery and warnings
2021-05-03 21:18:38 +00:00
CJ Cobb
4e58c76eac
Fix crashing bug related to account_tx writes
2021-05-03 21:18:36 +00:00
Nathan Nichols
ee763ec1ea
merge master
2021-05-03 12:13:50 -05:00
CJ Cobb
c0612e740e
Many changes around ledger_data and book_offers
...
* populate caches on startup
* implement new algorithms for Postgres
* use one shift interval instead of two
* block etl at flag ledger until writes are finished
* write one flag ledger ahead
* abandon ledger_diff component of ledger_data
2021-04-29 21:30:05 +00:00
CJ Cobb
f1ff81ddc5
postgres support for new books and keys tables
2021-04-29 14:51:03 +00:00
CJ Cobb
1f2d670e2b
Merge branch 'perf_test_clean' into master_new
2021-04-28 20:33:15 +00:00
CJ Cobb
b734884f20
various improvements around testing
2021-04-28 20:27:52 +00:00
CJ Cobb
8c6fe09590
compilation fix after merge
2021-04-28 20:16:47 +00:00
CJ Cobb
e747c61873
Merge branch 'ledger_data_indexer' into master_new
2021-04-28 20:12:28 +00:00
CJ Cobb
20184527af
Merge branch 'postgres_partitioning' into master_new
2021-04-28 19:55:25 +00:00
CJ Cobb
4349668000
Merge branch 'parallel_etl' into master_new
2021-04-28 19:55:08 +00:00
CJ Cobb
5ebc515978
support for read replicas with postgres
2021-04-28 19:02:44 +00:00
CJ Cobb
0cfce33724
drop hash index on objects.ledger_sequence. was making initial ingestion slow
2021-04-28 18:44:21 +00:00
CJ Cobb
40d136b3fc
expand primary key on account_transactions to support index only scans. fix account_tx stored procedure
2021-04-28 17:04:00 +00:00
CJ Cobb
3aed708262
make threadpool a class object
2021-04-27 16:19:45 +00:00
Nathan Nichols
3c8ecc01f1
impliment cursor for book_offers
2021-04-26 15:35:29 -05:00
CJ Cobb
3be12dd4b4
time account_tx_full
2021-04-26 14:35:34 +00:00
Nathan Nichols
0bcf3a4601
Order book offers by quality
2021-04-21 23:27:14 -05:00
CJ Cobb
1b7d672427
fix cursor issue with account_tx stored procedure
2021-04-20 17:20:35 +00:00
CJ Cobb
777dbb5826
account_tx stored procedure
2021-04-20 16:09:22 +00:00
CJ Cobb
e833bfb38c
simulate async fetch batch transactions
2021-04-16 18:53:05 +00:00
CJ Cobb
5364830224
fix postgres account_tx performance issue
2021-04-16 16:02:44 +00:00
CJ Cobb
d9a8ff5399
index during ETL. not tested
2021-04-15 14:10:28 +00:00
CJ Cobb
ed4ce93481
remove mention of keys table
2021-04-15 13:48:47 +00:00
CJ Cobb
d5f21ab295
fix indexes
2021-04-14 21:33:11 +00:00
CJ Cobb
23ba45ea5d
remove main tx hash index
2021-04-14 14:00:28 +00:00
CJ Cobb
dde44aa3e0
table partitioning
2021-04-12 21:14:29 +00:00
CJ Cobb
990e618fe8
add accum parameter
2021-04-12 15:27:16 +00:00
CJ Cobb
67f98e8ac9
fix memory leak
2021-04-08 20:09:21 +00:00
CJ Cobb
d773fcfdb7
add finish sequence
2021-04-07 13:43:24 +00:00
CJ Cobb
5f9e5d03f4
speed up indexer
2021-04-05 15:22:07 +00:00
CJ Cobb
4f834fc25f
fix memory leak
2021-04-01 23:43:54 +00:00
CJ Cobb
df8b6f15d9
make cassandra driver thread count configurable
2021-04-01 23:43:42 +00:00
CJ Cobb
0d11898730
indexer picks up from where it left off
2021-04-01 23:43:11 +00:00
CJ Cobb
db37c05b7b
new book offers algorithm, seems to work
2021-03-31 19:22:53 +00:00
CJ Cobb
7f39378268
test script updates
2021-03-31 15:57:02 +00:00
Ubuntu
d27b53e4f7
actually write to keys table
2021-03-30 19:36:55 +00:00
Ubuntu
ef555b00a7
indexer values look good. Nothing is being written yet
2021-03-30 18:35:40 +00:00
CJ Cobb
d1f47b490a
iterate through diffs. don't write anything
2021-03-30 11:13:51 -04:00
Ubuntu
2eca3cac96
enhance test.py
2021-03-26 19:41:52 +00:00
Ubuntu
8aaa365e46
bypass validated and hasLedger checks
2021-03-26 17:14:05 +00:00
CJ Cobb
9e1bc896e7
fix memory leak
2021-03-26 13:05:48 -04:00
CJ Cobb
58509093ac
make cassandra driver thread count configurable
2021-03-26 08:42:23 -04:00
CJ Cobb
d2da6051d1
reduce etl queue size based on number of extractor threads
2021-03-25 21:58:33 -04:00
CJ Cobb
168283f0aa
the best ledger_data implementation yet
2021-03-25 21:57:04 -04:00
CJ Cobb
2af6d72d7e
reduce etl queue size based on number of extractor threads
2021-03-25 21:57:04 -04:00
CJ Cobb
d72867ec64
switch deserialization order
2021-03-24 21:00:49 -04:00
CJ Cobb
29424fd863
remove sparsity algorithm in cassandra ledger page
2021-03-24 20:44:39 -04:00
Ubuntu
a77d13449d
update test.py and metrics.py
2021-03-25 00:42:44 +00:00