Commit Graph

  • 9b8e0bdf99 adjust complete query Nathan Nichols 2021-05-13 07:43:40 -05:00
  • f2c7d54b0d address pr feedback Nathan Nichols 2021-05-13 06:40:23 -05:00
  • d57cf3a518 add warning Nathan Nichols 2021-05-12 11:42:53 -05:00
  • 84911f9424 save for testing Nathan Nichols 2021-05-11 18:16:43 -05:00
  • d7ce14983a Impliments book_offers in postgres Nathan Nichols 2021-05-10 15:40:03 -05:00
  • 945598d953 merge master into branch Nathan Nichols 2021-05-10 15:10:58 -05:00
  • 3faf27fc4f impliment cassandra like fetch in postgres Nathan Nichols 2021-05-10 15:09:29 -05:00
  • f32af6bb2d various fixes. make shift info optional in config. Properly check book flag ledger is complete CJ Cobb 2021-05-07 20:49:42 +00:00
  • ce49de4fee impliment book offers for async etl Nathan Nichols 2021-05-06 23:07:52 -05:00
  • 144ad5eb19 allow different shifts for books and keys. default to 10 and 20 respectively CJ Cobb 2021-05-06 20:13:04 +00:00
  • 20b8059151 fix up some issues with async indexer CJ Cobb 2021-05-05 20:22:57 +00:00
  • 736e0a675f small fixes in CassandraBackend CJ Cobb 2021-05-03 18:54:32 +00:00
  • 64d0c5d050 async populate caches CJ Cobb 2021-04-30 20:15:38 +00:00
  • 971437f456 async write to keys and books with recovery and warnings CJ Cobb 2021-04-30 04:41:51 +00:00
  • 4e58c76eac Fix crashing bug related to account_tx writes CJ Cobb 2021-05-03 21:16:25 +00:00
  • ee763ec1ea merge master Nathan Nichols 2021-05-03 12:13:50 -05:00
  • c0612e740e Many changes around ledger_data and book_offers CJ Cobb 2021-04-29 21:30:05 +00:00
  • f1ff81ddc5 postgres support for new books and keys tables CJ Cobb 2021-04-29 14:51:03 +00:00
  • 1f2d670e2b Merge branch 'perf_test_clean' into master_new CJ Cobb 2021-04-28 20:33:15 +00:00
  • b734884f20 various improvements around testing CJ Cobb 2021-04-28 20:27:52 +00:00
  • 8c6fe09590 compilation fix after merge CJ Cobb 2021-04-28 20:16:47 +00:00
  • e747c61873 Merge branch 'ledger_data_indexer' into master_new CJ Cobb 2021-04-28 20:12:28 +00:00
  • 20184527af Merge branch 'postgres_partitioning' into master_new CJ Cobb 2021-04-28 19:55:25 +00:00
  • 4349668000 Merge branch 'parallel_etl' into master_new CJ Cobb 2021-04-28 19:55:08 +00:00
  • 5ebc515978 support for read replicas with postgres CJ Cobb 2021-04-28 19:02:44 +00:00
  • 0cfce33724 drop hash index on objects.ledger_sequence. was making initial ingestion slow CJ Cobb 2021-04-28 18:44:21 +00:00
  • 40d136b3fc expand primary key on account_transactions to support index only scans. fix account_tx stored procedure CJ Cobb 2021-04-28 15:35:20 +00:00
  • 3aed708262 make threadpool a class object CJ Cobb 2021-04-27 16:19:45 +00:00
  • 3c8ecc01f1 impliment cursor for book_offers Nathan Nichols 2021-04-26 15:35:29 -05:00
  • 3be12dd4b4 time account_tx_full CJ Cobb 2021-04-26 14:35:34 +00:00
  • 0bcf3a4601 Order book offers by quality Nathan Nichols 2021-04-21 23:27:14 -05:00
  • 1b7d672427 fix cursor issue with account_tx stored procedure CJ Cobb 2021-04-20 17:20:35 +00:00
  • 777dbb5826 account_tx stored procedure CJ Cobb 2021-04-19 20:07:55 +00:00
  • e833bfb38c simulate async fetch batch transactions CJ Cobb 2021-04-16 18:51:51 +00:00
  • 5364830224 fix postgres account_tx performance issue CJ Cobb 2021-04-13 20:00:47 +00:00
  • d9a8ff5399 index during ETL. not tested CJ Cobb 2021-04-08 20:07:05 +00:00
  • ed4ce93481 remove mention of keys table CJ Cobb 2021-04-15 13:48:47 +00:00
  • d5f21ab295 fix indexes CJ Cobb 2021-04-14 21:33:11 +00:00
  • 23ba45ea5d remove main tx hash index CJ Cobb 2021-04-14 14:00:28 +00:00
  • dde44aa3e0 table partitioning CJ Cobb 2021-04-12 21:14:29 +00:00
  • 990e618fe8 add accum parameter CJ Cobb 2021-04-12 15:27:16 +00:00
  • 67f98e8ac9 fix memory leak CJ Cobb 2021-03-26 12:55:07 -04:00
  • d773fcfdb7 add finish sequence CJ Cobb 2021-04-07 13:43:24 +00:00
  • 5f9e5d03f4 speed up indexer CJ Cobb 2021-04-05 15:22:07 +00:00
  • 4f834fc25f fix memory leak CJ Cobb 2021-03-26 12:55:07 -04:00
  • df8b6f15d9 make cassandra driver thread count configurable CJ Cobb 2021-03-26 08:42:23 -04:00
  • 0d11898730 indexer picks up from where it left off CJ Cobb 2021-04-01 23:43:11 +00:00
  • db37c05b7b new book offers algorithm, seems to work CJ Cobb 2021-03-31 19:22:53 +00:00
  • 7f39378268 test script updates CJ Cobb 2021-03-31 15:57:02 +00:00
  • d27b53e4f7 actually write to keys table Ubuntu 2021-03-30 19:36:55 +00:00
  • ef555b00a7 indexer values look good. Nothing is being written yet Ubuntu 2021-03-30 18:35:40 +00:00
  • d1f47b490a iterate through diffs. don't write anything CJ Cobb 2021-03-30 11:13:51 -04:00
  • 2eca3cac96 enhance test.py Ubuntu 2021-03-26 19:41:52 +00:00
  • 8aaa365e46 bypass validated and hasLedger checks Ubuntu 2021-03-26 17:14:05 +00:00
  • 9e1bc896e7 fix memory leak CJ Cobb 2021-03-26 12:55:07 -04:00
  • 58509093ac make cassandra driver thread count configurable CJ Cobb 2021-03-26 08:42:23 -04:00
  • d2da6051d1 reduce etl queue size based on number of extractor threads CJ Cobb 2021-03-25 21:56:03 -04:00
  • 168283f0aa the best ledger_data implementation yet CJ Cobb 2021-03-25 16:56:54 -04:00
  • 2af6d72d7e reduce etl queue size based on number of extractor threads CJ Cobb 2021-03-25 21:56:03 -04:00
  • d72867ec64 switch deserialization order CJ Cobb 2021-03-24 21:00:49 -04:00
  • 29424fd863 remove sparsity algorithm in cassandra ledger page CJ Cobb 2021-03-24 20:44:39 -04:00
  • a77d13449d update test.py and metrics.py Ubuntu 2021-03-25 00:42:44 +00:00
  • 0e23a2e0b5 update metrics.py to print aggregate at end CJ Cobb 2021-03-24 15:19:27 -04:00
  • fbd4a19f31 update metrics.py CJ Cobb 2021-03-24 15:06:35 -04:00
  • 59124e7b0a fix publish bug. check ledger range CJ Cobb 2021-03-24 15:00:11 -04:00
  • 14fd6d24ad add ancestry check CJ Cobb 2021-03-24 14:46:48 -04:00
  • c74f9654af inc current sequence in load thread CJ Cobb 2021-03-24 13:27:46 -04:00
  • 3062b5e678 add avg extract time CJ Cobb 2021-03-24 13:13:28 -04:00
  • 694111a9b7 numExtractors is config Ubuntu 2021-03-24 17:03:15 +00:00
  • 6eb87bfaff proper parallel extraction CJ Cobb 2021-03-24 05:33:27 -04:00
  • a7a71ec0c7 etl pipeline CJ Cobb 2021-03-24 04:29:54 -04:00
  • c69fb9672e update cmake version CJ Cobb 2021-03-24 01:54:45 -04:00
  • 564d46cb5f online delete with postgres. seems to work CJ Cobb 2021-03-23 15:55:09 -04:00
  • c6a42fa9f2 change log level of extract phase perf logging CJ Cobb 2021-03-23 10:58:43 -04:00
  • 2ef28ff9be enhance test.py and metrics.py CJ Cobb 2021-03-22 11:12:07 -04:00
  • 4fc44c5ac8 log parser CJ Cobb 2021-03-18 13:03:33 -04:00
  • 8d021f2019 don't cascade delete to objects and books CJ Cobb 2021-03-18 11:30:26 -04:00
  • aed0f8707f read only mode. ledger_data makes sure ledger is present first CJ Cobb 2021-03-18 11:20:02 -04:00
  • f476777ef5 fix book_offers postgres issue CJ Cobb 2021-03-17 15:34:45 -04:00
  • 6f06c60eef fix book offers paging CJ Cobb 2021-03-18 11:29:11 -04:00
  • d84c593064 add ledger entry handler (should have been added in prior commit) CJ Cobb 2021-03-12 17:31:08 -05:00
  • 9281ffb6e6 add cursor to offers response. verify offers work CJ Cobb 2021-03-12 17:30:50 -05:00
  • 4609010967 Add database timeout exception. clear out incomplete cassandra data. add ledger_entry rpc CJ Cobb 2021-03-12 16:37:27 -05:00
  • d2f0537f02 handle postgres timeouts CJ Cobb 2021-03-11 16:44:43 -05:00
  • d62f7accfe fix race condition in Postgres backend. add cursor to account_tx response. test script verifies account_tx CJ Cobb 2021-03-11 13:14:21 -05:00
  • b340f24e2c checkpoint CJ Cobb 2021-03-10 17:47:31 -05:00
  • c3b7983618 refactor CassandraBackend async bookkeeping CJ Cobb 2021-03-10 16:03:56 -05:00
  • f781bd6c00 ledger rpc accepts more flags. binary options for all rpcs. tests to verify ledger and account_info CJ Cobb 2021-03-09 16:33:24 -05:00
  • e3a121e571 bug fixes in both backends. add ledger_range rpc. improve test scripts CJ Cobb 2021-03-05 16:46:55 -05:00
  • 7a6dfe5967 postgres fixes CJ Cobb 2021-03-05 11:25:12 -05:00
  • 6bbb5579ca Ledger handler CJ Cobb 2021-03-04 14:34:55 -05:00
  • 694ec7bfe5 bug fixes. insert objects in batch CJ Cobb 2021-03-04 12:49:34 -05:00
  • 5606d4a7dd add type field to config. clean up cassandra code more CJ Cobb 2021-03-04 10:36:42 -05:00
  • 5771b31076 Merge branch 'backend_interface' CJ Cobb 2021-03-03 16:58:53 -05:00
  • db09ec23a8 don't write to keys table CJ Cobb 2021-03-03 16:50:46 -05:00
  • 6d39a5494f implement second version of fetchLedgerPage CJ Cobb 2021-03-03 16:48:00 -05:00
  • 997e3ac2e7 refactor of CassandraBackend. compiles, not tested CJ Cobb 2021-03-03 16:26:34 -05:00
  • 308c585801 partial refactor of CassandraBackend CJ Cobb 2021-03-02 16:19:14 -05:00
  • 8691cff8df rename CassandraFlatMapBackend to CassandraBackend and rename file CJ Cobb 2021-03-02 10:23:58 -05:00
  • 805bc14994 add cursor and limit to book offers CJ Cobb 2021-03-02 10:15:14 -05:00