Commit Graph

61 Commits

Author SHA1 Message Date
Savinda Senevirathne
16c5b3fae2 Introducing ledger shards and new ledger syncing logic. (#247)
- The ledgers are stored in a sqlite database in ledger file system.
- Ledgers are organized in shards inside primary folder.
- Raw inputs are saved as shards inside blob folder. No input files are created if neither inputs nor outputs are available.
- Both last primary shard hash and last blob shard hashes are subjected to consensus and necessary sync operations are performed if out of sync.
- Hpfs sync support setting single sync targets from both ends of the list. (Targets set from front are prioritized).
- Contract and ledger syncs and serving are performed independently.
2021-02-18 11:24:05 +05:30
Ravin Perera
9f4499653a Dynamic roundtime detection. (#244)
When consensus is unreliable detect roundtime based on roundtime reported by peers.
2021-02-16 13:34:29 +05:30
Savinda Senevirathne
a4399544b9 Separate contract and ledger file system mounts. (#235)
- Creating two separate hpfs folders for contract and ledger file systems (contract_fs and ledger_fs).
- Added separate mounts for contract and ledger file systems.
- Added separate instances for contract serving and ledger serving.
- Added separate instances for contract syncing and ledger syncing.
- Modified cluster creating script to accompany folder name changes in contract folder.
2021-02-05 11:48:52 +05:30
Savinda Senevirathne
d08d2630f6 Refactoring hpfs, hpfs sync and hpfs serve code. (#231)
* Refactoring hpfs code to a class so it can support multiple mounts.
* Refactoring hpfs serve into a class to support mulitiple mount serving.
* Refactoring hpfs sync into class to support multiple instances.
* Code improvements in hpfs_sync.
* Taking a sync target list for hpfs syncing target set.
2021-02-02 13:17:21 +05:30
Ravin Perera
70493d827d Contract id mismatch log message. (#227)
* Added contract id mismatch log.
* Added individual docker image run script.
2021-01-22 16:36:34 +05:30
Ravin Perera
610b67cec0 Contract execute config flag. (#226)
* Added contract execute flag. Made bin_path optional.
* Updated docker cluster default config.
* Renamed contract config to hp config.
2021-01-22 14:33:07 +05:30
Savinda Senevirathne
bed8205ca8 Config patch change detection. (#211)
* Applying patch file changes to hpcore runtime after patch file change detection.
* Removing unl sync functionality.
* Removing subjecting unl changeset to consensus.
2021-01-06 20:02:14 +05:30
Savinda Senevirathne
c87ae6693d Patch config and hpfs sync refactor. (#208)
* Terminology changes from state to hpfs.
* Syncing hpfs parents in priority order.
* Changing how the hash is calculate in hpfs response validate check.
2021-01-06 10:50:58 +05:30
Savinda Senevirathne
e835e18d18 New config file structure. (#201) 2020-12-23 18:52:58 +05:30
Savinda Senevirathne
0bc53fe23c Review UNL message rules. (#188) 2020-12-09 13:34:40 +05:30
Ravin Perera
fe9e276f8d Consensus deadlock fixes and reliability improvements. (#187)
* Consensus time vote improvements.
* Skipped self proposal in Observer mode.
* Added docker cluster script params.
* Added state/lcl sync abandon threshold.
* Added pubkey display for cluster script.
* Added proposal latency log.
* Added sync completion check to prevent deadlocks.
2020-12-08 15:17:27 +05:30
Savinda Senevirathne
7bf0475b6f Subject unl list to consensus. (#186)
* Unl hash in consensus proposals.
* Unl syncing and sync request serving.
* Adding unl hash to the ledger block.
2020-12-08 15:05:38 +05:30
Savinda Senevirathne
a421f13d91 Introduce consensus/npl public/private flags to config. (#182)
Npl messages are forwarded and broadcasted only to trusted nodes if npl flag is set to private. If public, messages are broadcasted to all the connected peers.
Consensus proposals are forwarded and broadcasted only to trusted nodes if consensus flag is set to private. If public, proposals are broadcasted to all the connected peers.
2020-12-03 08:12:56 +05:30
Savinda Senevirathne
f3055822ed Introduce pubkey/unl tracking to Peer sessions. (#181)
* Adding pubkey in binary to comm sessions and is_unl flag to peer sessions.
* Removing display_name overloads and populating pubkey in user session.
* Changing peer session lookup key from uniqueid to pubkey (binary).
2020-12-02 13:05:06 +05:30
Ravin Perera
596fd2b43c Added contract id verification. (#180)
* Added contract id config.
* Verify contract id with user/peer challenge.
* Updated cluster scripts.
* Updated node client lib for contract id verification.
2020-12-01 16:08:41 +05:30
Savinda Senevirathne
e90e9bb5dd Limiting NPL message broadcasting and forwarding to trusted nodes. (#173)
* Limiting NPL message broadcasting and forwarding to trusted nodes.
* Error fix in read_iosocket.
2020-11-30 11:11:40 +05:30
Ravin Perera
c41b37fd52 Request missing history when behind the ledger cap. (#165) 2020-11-25 16:11:46 +05:30
Chalith Desaman
459799760b Added element limits for queue and lists (#162) 2020-11-24 13:17:53 +05:30
Ravin Perera
d9517b6164 Reorganized util and hpws source files. (#161) 2020-11-23 23:18:42 +05:30
Savinda Senevirathne
fe24ef2388 Introduce peer requirement announcement. (#156) 2020-11-20 16:01:46 +05:30
Savinda Senevirathne
9160b2ee15 Improve corebill and graylisting. (#154) 2020-11-19 08:40:15 +05:30
Chalith Desaman
f475dcb1fb Dynamic peer list implementation. (#152) 2020-11-18 12:58:58 +05:30
Ravin Perera
3ea0299964 Comm session re-architecture and self comm channel. (#145)
* Introduced self comm channel instead of loopback websocket.
* Introduced comm_session and comm_server inheritance hierarchy.
* Separated peer session and user session classes.
2020-11-08 22:14:08 +05:30
Ravin Perera
ba0cae019d Refactored consensus into 3 rounds. (#144)
* Refactored consensus into 3 stages and removed stage 0.
* Consensus threshold calculation improvements.
* Refactored candidate user input processing.
* Renamed proposal sent timestamp field.
* Introduced comm_session display name.
2020-11-07 15:01:01 +05:30
Ravin Perera
202a6a2715 Ledger and state sync improvements. (#143)
* Increased stage time window.
* Introduced central state tracker.
* Used shared mutex for target state update.
* Refactored lcl and state sync checks in consensus.
* lcl/state sync loop wait time improvements.
2020-11-04 16:01:10 +05:30
Ravin Perera
37cc27c5ce Improved debug logging for state sync. (#139)
* Added random peer send pubkey extraction.
* Improved debug logs in state serve and state sync.
2020-10-27 16:03:34 +05:30
Savinda Senevirathne
fabfdcce89 Weakly connected status announcement. (#135)
* Forward others' messages only to the weakly connected nodes instead of broadcasting to all the connected peers.
* Announcing connected status depends on a threshold to other connected peers.
* Forwarding messages of weakly connected peers to other peers.
2020-10-23 16:57:01 +05:30
Savinda Senevirathne
4f6ff4fbef Peer message forwarding. (#133) 2020-10-19 10:44:31 +05:30
Ravin Perera
7183383ab7 Replace websocketd/websocat with hpws. (#131) 2020-10-15 17:02:06 +05:30
Chalith Desaman
5f40aebf08 NPL message refactor. (#132)
* Implemented feeding and broadcasting npl messages from the contract execution in real-time.
* Replaced npl pipe with domain sockets.
* Refactored npl read and write in nodejs echo contract
2020-10-14 15:18:00 +05:30
Ravin Perera
cb4d0c4f59 Ledger maintenance refactor. (#130)
* Added ledger namespace.
* Thread-safe lcl access and update.
* Refactored history sync and serving into a thread.
* Restructured ledger cache item.
2020-10-08 22:25:47 +05:30
priyadharsun
91122474a1 Replaced Boost log with Plog. (#124) 2020-09-23 06:06:42 +05:30
Savinda Senevirathne
38ef3cc1b1 Replacing lock_guard from scoped_lock. (#120) 2020-09-18 10:29:34 +05:30
Ravin Perera
d2f45daf4c Implemented multi-threaded inbound network message processing. (#115)
Used per-session thread to offload messages from network and a single thread for processing collected messages.
2020-09-10 15:40:08 +05:30
Ravin Perera
d4a786e3b9 Refactored NPL message processing. Passed lcl to contract args. (#105) 2020-08-05 21:30:48 +05:30
Ravin Perera
96f23cb0ff Added bson support for user message protocol. (#99) 2020-07-02 21:40:55 +05:30
Ravin Perera
04e230c82e Improvements in state request/response handling. (#97) 2020-06-18 15:07:33 +05:30
Ravin Perera
b89dbe0a2c Rearchitected state sync with hpfs. (#96) 2020-06-10 20:51:45 +05:30
Ravidu Lashan
ef8ec0f51d Added challenge issue and verification for peer connections (#93)
Implemented flatbuffer peer challenge issue and verification messages.
Replaced session flags with challenge status enum.
2020-04-20 05:55:42 +05:30
Ravin Perera
920be03ade Websocket re-architecture with websocketd and websocat (#89)
- Replaced beast with websocketd and websocat. #79 #83 #84
- Implemented inbound/outbound peer connection merging.
- Added graceful shutdown of hpcore with sigint. #87
2020-04-05 08:12:55 +05:30
Ravin Perera
4fefb7ca71 Refactored stage sync logic. (#86)
* Cleaned up stage time sync logic and avoided extra missed rounds.
* Moved stage sync time logic to beginning of consensus stage.
* Removed check_majority_stage.
* Re-organised lcl sync flow.
2020-02-10 14:27:37 +01:00
Asanka Indrajith
7a7aa6d5b3 Consensus enhancements and eliminating deadlocks after state inclusion. (#69) 2019-12-13 09:38:50 -05:00
Ravidu Lashan
1238e96423 State synchronization logic (#67)
* Added flat buffer state message request

* Added state vote

* Added state to ledger history and did necessary changes

* Completed receiveing state request

* State read/write helpers.

* Added new fbs schema

* Added more state_store helper methods.

* Started processing response

* Fixed compile errors

* Added get file length.

* Handled state content response

* Statefs code cleanup and fixes.

* Completed response handling

* Completed changes in handling state response

* State sync integration fixes.

* Fuse mount waiting logic.

* Fixed state syncing issues

* state sync fixes

* fixes

* State sync fixes.

* Fixed fs entries retrieval issues.

* changed desync logic

* Added directory helper functions.

* Handled return statemetns from statefs

* Fixed state folder deletion.

* handled errors from statefs

* Working for small files

* Got state sync working.

* Removed cout.

* Fixed catering for stae issue

* Fixed block hash map flatbuf issue.

* Added expected hash

* Added helpers for expected hash comparison.

* Improved state req/resp awaiting logic.

* Fixes.

* Fixes.

* Block request ordering fix.

* Removed couts

* Closed non-closed file descriptors

* Minor fixes.

* Cluster create script changes.

* Fixed reset time off issue.
2019-12-13 10:20:41 +05:30
Asanka Indrajith
95683035b9 LCL history request and response. (#59)
Detect and request missing lcl history from another random node.
Sending lcl history response to a asked node.
Getting lcl history response and applying it.
Delete lcl that exceeds max ledger sequence.
2019-11-19 07:50:33 -05:00
Ravidu Lashan
0439ec93e2 Added npl message feature (#58)
* Completed intial implementation

* Completed basic implementation of npl

* Completed implementation of npl

* Removed unused code

* completed review changes

* Removed unused code segments

* Added descriptive comments

* Added comment to describe npl message header
2019-11-15 10:56:35 +05:30
ravinsp
ef46666fd2 Added peer and user connection limits. 2019-11-10 14:55:55 +05:30
ravinsp
a89e7eb6c8 Reorganized p2p context objects. 2019-11-10 08:45:20 +05:30
Ravin
8d31075b7b Code improvements. 2019-11-07 17:21:21 +05:30
Ravin Perera
d16b43406b Added bad and duplicate message counters to socket_session (#55) 2019-11-07 06:57:34 +05:30
Ravin Perera
8a153d5bb5 Implemented user duplicate input message check. 2019-11-05 11:47:42 +05:30