Commit Graph

68 Commits

Author SHA1 Message Date
Ravin Perera
83bb158dd3 Fixed integer serialization issue with bson. (#287) 2021-04-12 07:15:57 +05:30
Ravin Perera
370e03fe84 Switched user input nonce from string to uint64. (#286) 2021-04-11 14:51:57 +05:30
Ravin Perera
3941f5f6c6 Revamped ledger input/output storage and query execution. (#285)
- Ledger sqlite db improvements.
- Removed flatbuffers from ledger blob data storage.
- Added consensus private mode blob data filtration.
2021-04-11 11:45:14 +05:30
Ravin Perera
0a3183a3d6 Included output hash and ledger info in output return message. (#284) 2021-04-08 10:42:33 +05:30
Ravin Perera
d4646179c2 Returned input hash and ledger info for input submissions. (#283)
- Introduced input hash which can later be used to query the ledger.
- Returned input hash and ledger info at input submission.
- Updated js client lib to support input hash.
- Updated consensus proposal candidate expiration rules.
2021-04-07 21:22:29 +05:30
Chalith Desaman
5833431157 hpfs binary update, hpfs log serving and passing to hpfs (#281)
* full history announcement in peer challenge and ability to send random messages only to full history nodes.

* log sync infrastructure codes.

* Resolving merge conflict  induced errors.

* Fork detection implementation.

* Cleanup.

* Resolving PR comments.

* hpfs binary update, hpfs log serving and persisting

* Fixed merge conflict and added missing log

Co-authored-by: Savinda Senevirathne <savindadilsara@gmail.com>
2021-04-01 17:13:58 +05:30
Ravin Perera
bba5266f5d Fixed consensus issues with large no. of users and inputs. (#280)
* Fixed issue in consensus contract output  sorting.

* Improved load test script.

* Removed flatbuffer verifier restrictions to support large no. of users and inputs.

* Improved consensus proposal cleanup.

* Restored the earlier proposal cleanup logic.

* Minor fixes.

* const fix.
2021-04-01 17:12:39 +05:30
Savinda Senevirathne
af48f3b01f Hpfs log file sync and fork detection. (#279)
Basic infrastructure for log file sync.
A fork is detected by comparing the hpfs log index file against the ledger.
Min and max ledger sequence hashes are sent to a full history node to request log history.
2021-04-01 13:08:29 +05:30
Ravin Perera
99c5912f14 Version management refactor. (#278)
- Introduced separate versioning for ledger file storage.
- Refactored versioning information code base.
2021-03-31 10:19:12 +05:30
Ravin Perera
ef42d4c986 Ledger query results raw input/output support. (#277) 2021-03-30 15:44:20 +05:30
Ravin Perera
948113398c Ledger query infrastructure. (#275)
* Added json ledger query param parser.

* Added initial query response creation.

* Updated client lib.

* Implemented get ledger by seq no.

* Added ledger query execution wrappers.

* Included log record info.

* Fixed empty output hash issue.

* Added bson support.

* Added db file existance check.

* Added requesy/reply tracking for queries in client lib.

* Improved multi connection usage in client lib.

* Added genesis ledger query support.

* Updated naming convention of query result fields.

* Comments.

* Used sqlite bind() for query param.

* Used binary hashes in ledger sqlite db.

* Missing const.
2021-03-29 11:20:15 +05:30
Savinda Senevirathne
b45cdb999d Include hp version number on all required places. (#265)
* Adding HP version header to binary files and adding hp table to ledger.sqlite.

* Updating hp_version in user challenge message.

* Minor improvement.
2021-03-10 20:08:04 +05:30
adheeb-adb
473fd4183c User message with more details (#263)
* user message with more details

* PR fixes
2021-03-08 12:25:29 +05:30
Chalith Desaman
c426e2a30e Accept proposals and npl messages from unl nodes only (#260)
* Skip proposal and npl messages from non unl nodes

* Minor code comment change

* Resolved PR comments and Changed weakly connected status check

* Changed weakly connected status counter

* Added a comment
2021-03-05 16:42:39 +05:30
Ravin Perera
1cebcb1c35 Config runtime re-save refactor. (#258)
* Check peer discovery config when persisting peers.

* Refactored config persistance.

* Fixed config read offset issue.

* Refactored peer list structs.

* Minor constexpr.
2021-02-25 14:25:12 +05:30
Ravin Perera
5b56d9c1b3 Updated hpfs hash verification with file mode checks. (#257)
* Added file mode to hpfs responses.

* Parse file mode on hpfs response read.

* Apply file/dir mode.

* Fixed mode apply logic.

* Fixed code review comments.

* Additional fix.
2021-02-24 11:13:28 +05:30
Chalith Desaman
363116fc2a Added user input max ledger seq offset limit config (#256)
- Added new config for max input ledger offset inside contract section.
- Updated implementation of getting and updating patch config inside node and c contracts.
- Skip inputs if max ledger offset exceeds, when handling user inputs and nup messages.
2021-02-24 09:19:36 +05:30
Ravin Perera
137c199b07 P2P Flatbuffers schema refactor. (#255) 2021-02-23 13:15:03 +05:30
Ravin Perera
6b8d60a404 Separated lcl string usage to sequence no and hash. (#251) 2021-02-18 18:28:38 +05:30
Chalith Desaman
7059f68f11 Removed legacy lcl code and dependencies (#249) 2021-02-18 17:07:50 +05:30
Savinda Senevirathne
16c5b3fae2 Introducing ledger shards and new ledger syncing logic. (#247)
- The ledgers are stored in a sqlite database in ledger file system.
- Ledgers are organized in shards inside primary folder.
- Raw inputs are saved as shards inside blob folder. No input files are created if neither inputs nor outputs are available.
- Both last primary shard hash and last blob shard hashes are subjected to consensus and necessary sync operations are performed if out of sync.
- Hpfs sync support setting single sync targets from both ends of the list. (Targets set from front are prioritized).
- Contract and ledger syncs and serving are performed independently.
2021-02-18 11:24:05 +05:30
Ravin Perera
e394497698 Used uint32 for roundtime and timeout fields. (#248) 2021-02-17 12:21:51 +05:30
Ravin Perera
90641e0849 Persist discovered peers to config. (#246) 2021-02-16 16:20:19 +05:30
Ravin Perera
9f4499653a Dynamic roundtime detection. (#244)
When consensus is unreliable detect roundtime based on roundtime reported by peers.
2021-02-16 13:34:29 +05:30
Ravin Perera
033b5fa7bc User inputs round limit. (#240) 2021-02-14 20:19:53 +05:30
Savinda Senevirathne
a4399544b9 Separate contract and ledger file system mounts. (#235)
- Creating two separate hpfs folders for contract and ledger file systems (contract_fs and ledger_fs).
- Added separate mounts for contract and ledger file systems.
- Added separate instances for contract serving and ledger serving.
- Added separate instances for contract syncing and ledger syncing.
- Modified cluster creating script to accompany folder name changes in contract folder.
2021-02-05 11:48:52 +05:30
Savinda Senevirathne
d08d2630f6 Refactoring hpfs, hpfs sync and hpfs serve code. (#231)
* Refactoring hpfs code to a class so it can support multiple mounts.
* Refactoring hpfs serve into a class to support mulitiple mount serving.
* Refactoring hpfs sync into class to support multiple instances.
* Code improvements in hpfs_sync.
* Taking a sync target list for hpfs syncing target set.
2021-02-02 13:17:21 +05:30
Chalith Desaman
eef6448f08 Handle unl change announcement in js client library. (#225) 2021-01-19 15:29:17 +05:30
Chalith Desaman
73f5eea451 Unl change announcement to connected users. (#224)
Introduced json and bson messages for unl list announcement.
When the unl set is modified send a json or bson unl list message (according to the user protocol) to all the connected users.
2021-01-19 11:33:27 +05:30
Savinda Senevirathne
bed8205ca8 Config patch change detection. (#211)
* Applying patch file changes to hpcore runtime after patch file change detection.
* Removing unl sync functionality.
* Removing subjecting unl changeset to consensus.
2021-01-06 20:02:14 +05:30
Savinda Senevirathne
c87ae6693d Patch config and hpfs sync refactor. (#208)
* Terminology changes from state to hpfs.
* Syncing hpfs parents in priority order.
* Changing how the hash is calculate in hpfs response validate check.
2021-01-06 10:50:58 +05:30
Ravin Perera
f97d7de530 Moved h32 hash type from hpfs to util. (#204) 2020-12-23 21:58:49 +05:30
Ravin Perera
c25ebe66f4 Util helper func simplification. (#203) 2020-12-23 21:49:13 +05:30
Ravin Perera
8dc20bdab0 Naming changes and reintroducing key prefix to contract libraries. (#202) 2020-12-23 19:23:46 +05:30
Savinda Senevirathne
e835e18d18 New config file structure. (#201) 2020-12-23 18:52:58 +05:30
Ravin Perera
1a954ad96e Implemented output signature checks. (#200) 2020-12-22 16:55:44 +05:30
Chalith Desaman
99668be944 Added version to the ledger file structure. (#195) 2020-12-15 13:01:11 +05:30
Ravin Perera
f2ed9040c0 User protocol upgrade and js client lib. (#191)
* Unified js client lib for browser and nodejs.
* Client lib multiple connections support.
* Implemented server challenge response.
* Contract guid and version validation.
* Server key validation.
* User json message encoding improvements.
2020-12-11 11:02:58 +05:30
Savinda Senevirathne
0bc53fe23c Review UNL message rules. (#188) 2020-12-09 13:34:40 +05:30
Ravin Perera
fe9e276f8d Consensus deadlock fixes and reliability improvements. (#187)
* Consensus time vote improvements.
* Skipped self proposal in Observer mode.
* Added docker cluster script params.
* Added state/lcl sync abandon threshold.
* Added pubkey display for cluster script.
* Added proposal latency log.
* Added sync completion check to prevent deadlocks.
2020-12-08 15:17:27 +05:30
Savinda Senevirathne
7bf0475b6f Subject unl list to consensus. (#186)
* Unl hash in consensus proposals.
* Unl syncing and sync request serving.
* Adding unl hash to the ledger block.
2020-12-08 15:05:38 +05:30
Chalith Desaman
a87e8a0c7e Subject UNL changeset to consensus. (#183) 2020-12-07 13:06:58 +05:30
Ravin Perera
b2fd8ae4b5 Contract and client library improvements. (#184)
* Added tty check to contract libs.
* Javascript browser-native client.
* Removed hex encoding in user json outputs.
* Updated file contract for new contract library.
2020-12-05 09:08:24 +05:30
Ravin Perera
596fd2b43c Added contract id verification. (#180)
* Added contract id config.
* Verify contract id with user/peer challenge.
* Updated cluster scripts.
* Updated node client lib for contract id verification.
2020-12-01 16:08:41 +05:30
Chalith Desaman
2f5d95ae7a Save raw inputs alongside the ledger (#167)
* Saving raw inputs in the ledger.
* Store full history in separate folder.

* Moved full history fb structs to a seperate fbs file

* Removed raw_inputs from the proposal

* Fixed code comments

* Saving raw user input string

* Code refactoring

* Resolved PR comments
2020-11-30 15:42:20 +05:30
Ravin Perera
332e5a4750 User input json/bson format improvements. (#174)
* Removed hex encoding in json input container.
* Refactored client lib exports.
2020-11-28 21:24:35 +05:30
Ravin Perera
4de1bb2393 Already submitted input detection. (#170)
* Added same nonce/sig for input comparison.
* Added status response withholding for already submitted inputs.
2020-11-26 23:27:40 +05:30
Ravin Perera
d476f787a7 Introduced UNL update control message. (#169) 2020-11-26 22:26:12 +05:30
Ravin Perera
bb223b1e3a JSON message parsing for contract control messages. (#166) 2020-11-26 09:31:39 +05:30
Ravin Perera
c41b37fd52 Request missing history when behind the ledger cap. (#165) 2020-11-25 16:11:46 +05:30