100 Commits

Author	SHA1	Message	Date
CJ Cobb	f1ff81ddc5	postgres support for new books and keys tables	2021-04-29 14:51:03 +00:00
CJ Cobb	8c6fe09590	compilation fix after merge	2021-04-28 20:16:47 +00:00
CJ Cobb	e747c61873	Merge branch 'ledger_data_indexer' into master_new	2021-04-28 20:12:28 +00:00
CJ Cobb	20184527af	Merge branch 'postgres_partitioning' into master_new	2021-04-28 19:55:25 +00:00
CJ Cobb	5ebc515978	support for read replicas with postgres	2021-04-28 19:02:44 +00:00
CJ Cobb	0cfce33724	drop hash index on objects.ledger_sequence. was making initial ingestion slow	2021-04-28 18:44:21 +00:00
CJ Cobb	40d136b3fc	expand primary key on account_transactions to support index only scans. fix account_tx stored procedure	2021-04-28 17:04:00 +00:00
CJ Cobb	3aed708262	make threadpool a class object	2021-04-27 16:19:45 +00:00
CJ Cobb	1b7d672427	fix cursor issue with account_tx stored procedure	2021-04-20 17:20:35 +00:00
CJ Cobb	777dbb5826	account_tx stored procedure	2021-04-20 16:09:22 +00:00
CJ Cobb	e833bfb38c	simulate async fetch batch transactions	2021-04-16 18:53:05 +00:00
CJ Cobb	5364830224	fix postgres account_tx performance issue	2021-04-16 16:02:44 +00:00
CJ Cobb	d9a8ff5399	index during ETL. not tested	2021-04-15 14:10:28 +00:00
CJ Cobb	ed4ce93481	remove mention of keys table	2021-04-15 13:48:47 +00:00
CJ Cobb	d5f21ab295	fix indexes	2021-04-14 21:33:11 +00:00
CJ Cobb	23ba45ea5d	remove main tx hash index	2021-04-14 14:00:28 +00:00
CJ Cobb	dde44aa3e0	table partitioning	2021-04-12 21:14:29 +00:00
CJ Cobb	990e618fe8	add accum parameter	2021-04-12 15:27:16 +00:00
CJ Cobb	d773fcfdb7	add finish sequence	2021-04-07 13:43:24 +00:00
CJ Cobb	5f9e5d03f4	speed up indexer	2021-04-05 15:22:07 +00:00
CJ Cobb	4f834fc25f	fix memory leak	2021-04-01 23:43:54 +00:00
CJ Cobb	df8b6f15d9	make cassandra driver thread count configurable	2021-04-01 23:43:42 +00:00
CJ Cobb	0d11898730	indexer picks up from where it left off	2021-04-01 23:43:11 +00:00
CJ Cobb	db37c05b7b	new book offers algorithm, seems to work	2021-03-31 19:22:53 +00:00
Ubuntu	d27b53e4f7	actually write to keys table	2021-03-30 19:36:55 +00:00
Ubuntu	ef555b00a7	indexer values look good. Nothing is being written yet	2021-03-30 18:35:40 +00:00
CJ Cobb	d1f47b490a	iterate through diffs. don't write anything	2021-03-30 11:13:51 -04:00
Ubuntu	8aaa365e46	bypass validated and hasLedger checks	2021-03-26 17:14:05 +00:00
CJ Cobb	9e1bc896e7	fix memory leak	2021-03-26 13:05:48 -04:00
CJ Cobb	58509093ac	make cassandra driver thread count configurable	2021-03-26 08:42:23 -04:00
CJ Cobb	d2da6051d1	reduce etl queue size based on number of extractor threads	2021-03-25 21:58:33 -04:00
CJ Cobb	168283f0aa	the best ledger_data implementation yet	2021-03-25 21:57:04 -04:00
CJ Cobb	2af6d72d7e	reduce etl queue size based on number of extractor threads	2021-03-25 21:57:04 -04:00
CJ Cobb	d72867ec64	switch deserialization order	2021-03-24 21:00:49 -04:00
CJ Cobb	29424fd863	remove sparsity algorithm in cassandra ledger page	2021-03-24 20:44:39 -04:00
CJ Cobb	59124e7b0a	fix publish bug. check ledger range	2021-03-24 15:00:11 -04:00
CJ Cobb	14fd6d24ad	add ancestry check	2021-03-24 14:46:48 -04:00
CJ Cobb	c74f9654af	inc current sequence in load thread	2021-03-24 13:27:46 -04:00
CJ Cobb	3062b5e678	add avg extract time	2021-03-24 13:13:28 -04:00
Ubuntu	694111a9b7	numExtractors is config	2021-03-24 17:03:15 +00:00
CJ Cobb	6eb87bfaff	proper parallel extraction	2021-03-24 05:33:27 -04:00
CJ Cobb	a7a71ec0c7	etl pipeline	2021-03-24 04:29:54 -04:00
CJ Cobb	564d46cb5f	online delete with postgres. seems to work	2021-03-23 15:55:09 -04:00
CJ Cobb	c6a42fa9f2	change log level of extract phase perf logging	2021-03-23 10:58:43 -04:00
CJ Cobb	2ef28ff9be	enhance test.py and metrics.py	2021-03-22 11:16:10 -04:00
CJ Cobb	4fc44c5ac8	log parser	2021-03-18 13:05:47 -04:00
CJ Cobb	8d021f2019	don't cascade delete to objects and books	2021-03-18 11:34:20 -04:00
CJ Cobb	aed0f8707f	read only mode. ledger_data makes sure ledger is present first	2021-03-18 11:34:20 -04:00
CJ Cobb	f476777ef5	fix book_offers postgres issue	2021-03-18 11:34:20 -04:00
CJ Cobb	6f06c60eef	fix book offers paging	2021-03-18 11:34:20 -04:00