About S&D Ingest

S&D INGEST it's the module that receives row files in different formats and send's them to any module after the file's are being processed.

🌀 Server Requirements:

• php7.4 [https://www.php.net] LICENSE
• apache [https://httpd.apache.org] LICENSE
• redis [https://redis.io] LICENSE
• postgresql-server [https://www.postgresql.org] LICENSE
• supervisor [http://supervisord.org] LICENSE
• libraoffice [https://www.libreoffice.org] LICENSE
• python [https://www.python.org/] LICENSE
• pdftotext [https://github.com/jalan/pdftotext] LICENSE

⚡ Build with:

• Laravel Framework ^6.2

🚀 Installation

apt-get update
apt-get install software-properies-common
add-apt-repository ppa:deadsnakes/ppa
apt-get install supervisor python3.8 python3.8-dev
apt-get install redis-server

supervisorctl restart all
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
rm -rf get-pip.py
apt install libpoppler-cpp-dev
pip install --upgrade pip
pip install pdftotext supervisor
systemctl enable supervisor
php artisan queue:deploy-supervisor
systemctl restart supervisor
composer install
npm install
cp .env.example .env
php artisan key:generate
sudo -u postgres psql
postgres=# create database mydb;
postgres=# create user myuser with encrypted password 'mypass';
postgres=# grant all privileges on database mydb to myuser;
#update the .env with the current postgres credentials
sudo mkdir /var/log/amqp
sudo mkdir /var/log/queue
php artisan migrate
php artisan queue:deploy-supervisor
supervisorctl start all

Local Usage

php artisan serve
php artisan queue:work

PHP Packages

• fideloper/proxy LICENSE
• laravel/framework LICENSE
• laravel/tinker LICENSE
• phpoffice/phpword LICENSE
• predis/predis LICENSE
• spatie/laravel-webhook-server LICENSE

Current running process

• [DOC,DOCX,RTF etc..] are first being converted to docx and then converted to .txt using https://www.libreoffice.org
• [PDF] files are converted to .txt
• The resulting .txt file is processed using our own logic/alogorithm and clause breaking point to an array that looks similar to this:

 {
        "content": "Definitions and Interpretation",
        "spaces": 4,
        "numbering": "1.",
        "children": [
            {
                "content": "In this Agreement, the following expressions shall have the meanings set opposite them, unless inconsistent with the context or otherwise specified:",
                "spaces": 8,
                "numbering": "1.1",
                "children": [
                    {
                        "content": "“Agreement” this agreement including all schedules, appendices and exhibits attached herein;",
                        "spaces": 0
                    },
                    {
                        "content": "“Associated Company” any company which is listed in Schedule 2 (as may be amended from time to time in writing) and which is in relation to either party its Parent undertaking or its subsidiary undertaking or a subsidiary undertaking of its Parent undertaking or any other person controlled by it or under the same control (where “control” is to be construed in accordance with section 1124 of the Corporation Tax Act 2010) whether direct or indirect.  “Parent undertaking” shall have the meaning attributed thereto in Section 1162 of the Companies Act 2006;",
                        "spaces": 0
                    },
                    {
                        "content": "“Commencement Date” [TBC] “Confidential Information” collectively and individually, all or any document or information of any nature in any format, including oral, written or electronic form relating to either party or their Associated Companies’ or either of their businesses, including technology, Customers, Customer Information, supplier, employees, finances, data, products, services, trade secrets, processes, designs, drawings, diagrams, plans, specifications, formulae, testing procedures, computer software, reports, investigative studies, manuals, assets, costs, prices, marketing opportunities, proprietary information, Know-how, the terms of this Agreement and any other information or material relating to the information described above which (i) is disclosed by either party (or by any individual or legal entity acting in their name or on their behalf, including employees, consultants, sub-contractors, advisors of any kind and agents) or (ii) which comes to the attention of either party (or any individual or legal entity acting in their name or on their behalf, including employees, consultants, sub-contractors, advisors of any kind and agents) during the course of the carrying out of the rights or obligations under this Agreement;",
                        "spaces": 0
                    },
                    {
                        "content": "“Customers” customers of {P1_Name} and/or its Associated Companies from time to time who owe a Debt to {P1_Name};",
                        "spaces": 0
                    },
                    {
                        "content": "“Customer Data” any information given by the Customer directly to the {P2_Name} or its personnel;",
                        "spaces": 0
                    },
                    {
                        "content": "“Customer Information” any Customers’ personal information supplied to the {P2_Name} by or on behalf of {P1_Name} during the performance of this Agreement, including Personal Data, but excluding Customer Data;",
                        "spaces": 0
                    },
                    {
                        "content": "“Data Protection Legislation” all applicable legislation concerning the protection of individuals with regard to the processing of Personal Data and the free movement of such data including the Data Protection Act 1998 and any regulations made under such legislation and any relevant codes of practice and guidance notes issued from time to time by the Information Commissioner (or its successor);",
                        "spaces": 0
                    },
                    {
                        "content": "“Debt” any monies owed by the Customer to {P1_Name} which have remained unpaid by the Customer contrary to the terms and conditions between {P1_Name} and the Customer governing the repayment of sums owed;",
                        "spaces": 0
                    },
                    {
                        "content": "“Debt Management Plan” a plan outlined by the {P2_Name} and agreed by the Customer which details the amount and frequency of payments to be made to each of the Customer’s creditors;",
                        "spaces": 0
                    },
                    {
                        "content": "“Disbursement” total amount remitted to {P1_Name} on a monthly basis by the {P2_Name} for application to the Customer’s account in respect of their Offer;",
                        "spaces": 0
                    },
                    {
                        "content": "“European Economic Area” the European Economic Area comprising of the following countries as at the Commencement Date: Austria; Belgium; Bulgaria; Cyprus; the Czech Republic; Denmark; Estonia; Finland; France; Germany; Greece; Hungary; Ireland; Italy; Latvia; Lithuania; Luxembourg; Malta; the Netherlands; Poland; Portugal; Romania; Slovakia; Slovenia; Spain; Sweden; the United Kingdom; Iceland; Liechtenstein and Norway, as amended from time to time;",
                        "spaces": 0
                    },
                    {
                        "content": "“EU Model Terms” the set of model contractual clauses which the Information Commissioner has authorised for use by Data Controllers (as such term is defined in the Data Protection Legislation) established in the European Union where there is a transfer of Personal Data to Data Processors (as such term is defined in the Data Protection Legislation) outside of the European Economic Area;",
                        "spaces": 0
                    },
                    {
                        "content": "“Facility” the {P2_Name} site authorised by {P1_Name} where the processing and/or storage of Personal Data supplied by {P1_Name} pursuant to this Agreement takes place.  For the purposes of this Agreement that site shall be located at {P1_Reg} or such other place as may be notified in writing to {P1_Name} from time to time;",
                        "spaces": 0
                    },
                    {
                        "content": "“Force Majeure” any acts, events, omissions or accidents beyond the reasonable control of either Party, including but not limited to acts of God, extreme adverse weather conditions or natural disaster, war, threat of or preparation for war, armed conflict, imposition of sanctions, embargo, breaking off of diplomatic relations or similar actions, terrorist attack, civil war, civil commotion or riots, nuclear, chemical or biological contamination or sonic boom, compliance with any law, regulation or directive, fire, explosion or accidental damage, failure of plant machinery, machinery, computers or vehicles;",
                        "spaces": 0
                    },
                    {
                        "content": "“Information Commissioner” the independent authority in the UK (or its successor body) which regulates information rights;",
                        "spaces": 0
                    },
                    {
                        "content": "“Initial Period” three (3) years from the Commencement Date;",
                        "spaces": 0
                    },
                    {
                        "content": "“Lending Code” a voluntary code of practice (enforced by the Lending Standards Board) which sets standards for financial institutions and provides consumers with protection and explanation on how such institutions are expected to deal with them day-to-day and in times of financial difficulties;",
                        "spaces": 0
                    },
                    {
                        "content": "“Notification” written notification from the {P2_Name} to {P1_Name} that it has obtained Permission;",
                        "spaces": 0
                    },
                    {
                        "content": "“Offer” a statement of proposed amount to be repaid by the Customer to {P1_Name} in respect of the Customer’s Debt including instalment plans;",
                        "spaces": 0
                    },
                    {
                        "content": "“Payment Break” instance where the Customer fails to make an agreed repayment to the {P2_Name} for payment to their creditors;",
                        "spaces": 0
                    },
                    {
                        "content": "“Permission” written confirmation (which may be confirmation by email or other electronic means) from the Customer to the {P2_Name} that they are appointing the {P2_Name} to act on the Customer’s behalf in the management of the Customer’s Debt and authorising the {P2_Name} to negotiate payment terms with {P1_Name} in respect of the Customer’s Debt and authorising the {P2_Name} to have access to Customer Information;",
                        "spaces": 0
                    },
                    {
                        "content": "“Personal Data” personal data as defined in the Data Protection Legislation;",
                        "spaces": 0
                    },
                    {
                        "content": "“Regulatory Authorities” any body who, from time to time, has competent rule-making, investigatory and/or enforcement powers in relation to the business of {P1_Name} and/or its Associated Companies, including, without limitation, the Financial Conduct Authority, the Consumer Financial Protection Bureau, the Office of Fair Trading, the Information Commissioner’s Office, the Lending Standards Board, UK and US Government departments and organisations, the Office of the Comptroller of Currency, the Federal Reserve and other governmental or non-governmental regulatory authorities in the UK, US or other competent jurisdictions;",
                        "spaces": 0
                    },
                    {
                        "content": "“Regulatory Requirements”",
                        "spaces": 0
                    },
                    {
                        "content": "(a) all applicable laws, statutes, regulations, ordinances or subordinate legislation in force from time to time to which this Agreement or a party is subject;",
                        "spaces": 4
                    },
                    {
                        "content": "(b) the common law as applicable to the parties from time to time;",
                        "spaces": 4
                    },
                    {
                        "content": "(c) all binding court orders, judgements or decrees;",
                        "spaces": 4
                    },
                    {
                        "content": "all applicable directives, policies, rules, orders, code of conduct or practice or applicable guidance (including the Lending Code and the Financial  Conduct Authority TCF principles  that are binding on a party and that are made or given by any government, an agency thereof, any Regulatory Authority or other regulatory authority, including in the case of the {P2_Name},  laws and rules imposed by local regulatory authorities in the country where it is located;",
                        "spaces": 0
                    },
                    {
                        "content": "“Working Day” any day on which banks in London are open for the transaction of normal business excluding Saturdays, Sundays and bank and public holidays in England and Wales.",
                        "spaces": 0
                    }
                ]
            },
            {
                "content": "In this Agreement:",
                "spaces": 8,
                "numbering": "1.2",
                "children": [
                    {
                        "content": "references to Recitals, Clauses and Schedules and their sub-divisions are to the Recitals to, Clauses of and Schedules to this Agreement and their sub-divisions respectively, unless specified otherwise;",
                        "spaces": 12,
                        "numbering": "1.2.1"
                    },
                    {
                        "content": "the index and headings are included for convenience only and shall not affect the construction or interpretation of this Agreement;",
                        "spaces": 12,
                        "numbering": "1.2.2"
                    },
                    {
                        "content": "words importing gender include the other gender and the singular includes the plural and vice versa;",
                        "spaces": 12,
                        "numbering": "1.2.3"
                    },
                    {
                        "content": "references to persons include individuals, bodies corporate, firms, unincorporated associations and governmental, semi-governmental and local authorities or agencies;",
                        "spaces": 12,
                        "numbering": "1.2.4"
                    },
                    {
                        "content": "references to the words “include”, “including”, “in particular” or similar words or expressions will be construed without limitation and accordingly will not limit the words preceding or following them;",
                        "spaces": 12,
                        "numbering": "1.2.5"
                    },
                    {
                        "content": "where expressions used in this Agreement are not specifically defined and are capable of having a special meaning according to the usage or custom of the card services sector or banking services sector, such expressions are to be interpreted accordingly.  Any meaning given in this Agreement to a defined term shall prevail over such other special meaning;",
                        "spaces": 12,
                        "numbering": "1.2.6"
                    },
                    {
                        "content": "references to a “party” or “parties” will mean either {P1_Name} and/or the {P2_Name} as the context requires and references to a third party will mean any person other than the parties;",
                        "spaces": 12,
                        "numbering": "1.2.7"
                    },
                    {
                        "content": "except where expressly stated otherwise, references to any statute, legislation, code of practice or other regulation will include any sub-ordinate legislation and any equivalent regulation in any relevant jurisdiction, as amended, modified, consolidated, re-enacted and/or replaced and in force from time to time;",
                        "spaces": 12,
                        "numbering": "1.2.8"
                    },
                    {
                        "content": "any negative obligation imposed on any party shall be construed as if it were also an obligation not to permit or suffer the act or thing in question and any positive obligation imposed on any party shall be construed as if it were also an obligation to procure that the act or thing in question be done;",
                        "spaces": 12,
                        "numbering": "1.2.9"
                    },
                    {
                        "content": "the Schedules and Appendices (if any) form part of this Agreement and shall be construed and have the same full force and effect as if expressly set out in the body of this Agreement. To the extent only of any conflict or inconsistency between the Clauses, Schedules and Appendices (if any), the Clauses will prevail and the order of precedence will be as follows:",
                        "spaces": 12,
                        "numbering": "1.2.10"
                    },
                    {
                        "content": "1 the provisions of the Clauses;",
                        "spaces": 16,
                        "numbering": "1.2.10"
                    },
                    {
                        "content": "2 the provisions of the Schedules; and",
                        "spaces": 16,
                        "numbering": "1.2.10"
                    },
                    {
                        "content": "3 the provisions of the Appendices.",
                        "spaces": 16,
                        "numbering": "1.2.10"
                    }
                ]
            }
        ]
    },
    {
        "content": "Obligations of the {P2_Name}",
        "spaces": 4,
        "numbering": "2.",
        "children": [
            {
                "content": "The {P2_Name} shall obtain the Permission from the Customer before proceeding with the Debt Management Plan.",
                "spaces": 8,
                "numbering": "2.1"
            },
            {
                "content": "Subject at all times to the {P2_Name} being in receipt of the applicable Permission, {P2_Name} shall provide the corresponding Notification to {P1_Name} before or at the time of making the first Offer to {P1_Name}. In the absence of such Permission or Notification {P1_Name} shall not be obliged to provide any Customer Information to the {P2_Name}.",
                "spaces": 8,
                "numbering": "2.2"
            },
            {
                "content": "{P1_Name} may request, and the {P2_Name} shall provide, any Permission to {P1_Name} within two (2) Working Days of such request by {P1_Name} to enable {P1_Name} to verify the Permissions stated in the Notifications provided that in the event that {P1_Name} requests ten (10) or more Permissions in any 12 hour period then the {P2_Name} shall provide such Permissions as promptly as is reasonably possible.",
                "spaces": 8,
                "numbering": "2.3"
            },
            {
                "content": "Any delay or failure by the {P2_Name} to comply with Clause 2.3 shall be deemed a material breach of this Agreement and the provisions of clause 9.3 shall apply.",
                "spaces": 8,
                "numbering": "2.4"
            },
            {
                "content": "Subject to Clause 2.1 and in accordance with the Debt Management Plan, the {P2_Name} shall make an Offer to {P1_Name} for the repayment of the Debt detailing the amount and frequency of proposed payments. Such Offer will be made in accordance with the Lending Code guidelines and based upon the principle of equitable distribution of available income (after priority payments) in line with the amount outstanding to each creditor.",
                "spaces": 8,
                "numbering": "2.5"
            },
            {
                "content": "Upon receipt of the Offer from the {P2_Name} {P1_Name} may either;",
                "spaces": 8,
                "numbering": "2.6",
                "children": [
                    {
                        "content": "accept the Offer; or",
                        "spaces": 12,
                        "numbering": "2.6.1"
                    },
                    {
                        "content": "reject the Offer where it considers the offer to be unreasonable by written notice to the {P2_Name}.",
                        "spaces": 12,
                        "numbering": "2.6.2"
                    }
                ]
            },
            {
                "content": "In the event that {P1_Name} accepts an Offer, then the {P2_Name} shall arrange for the Disbursement to be repaid to {P1_Name} in accordance with the Offer within five (5) Working Days of receipt by the {P2_Name} of cleared funds from the Customer.",
                "spaces": 8,
                "numbering": "2.7"
            },
            {
                "content": "In the event that {P1_Name} rejects the Offer, then the {P2_Name} shall review the Debt Management Plan and the {P2_Name} may make a new Offer to {P1_Name}.",
                "spaces": 8,
                "numbering": "2.8"
            },
            {
                "content": "For the avoidance of doubt nothing in this Agreement constitutes an obligation on {P1_Name} to accept any unreasonable Offer made by the {P2_Name}.",
                "spaces": 8,
                "numbering": "2.9"
            },
            {
                "content": "The {P2_Name} shall notify {P1_Name} in writing as soon as reasonably possible:",
                "spaces": 8,
                "numbering": "2.10",
                "children": [
                    {
                        "content": "upon becoming aware of any withdrawal of a Permission or any amendment thereto made by a Customer; and",
                        "spaces": 12,
                        "numbering": "2.10.1"
                    },
                    {
                        "content": "of any circumstance or event which is reasonably likely to materially affect the {P2_Name}’s ability to comply with its obligations under this Agreement.",
                        "spaces": 12,
                        "numbering": "2.10.2"
                    }
                ]
            },
            {
                "content": "Failure by the {P2_Name} to notify {P1_Name} pursuant to Clause 2.10.1 shall be deemed a material breach of this Agreement and the provisions of clause 9.3 shall apply.",
                "spaces": 8,
                "numbering": "2.11"
            },
            {
                "content": "The {P2_Name} shall;",
                "spaces": 8,
                "numbering": "2.12",
                "children": [
                    {
                        "content": "at all times act in accordance with and subject to any limitations set out in (i) the Permission and (ii) the requirements of this Agreement;",
                        "spaces": 12,
                        "numbering": "2.12.1"
                    },
                    {
                        "content": "comply with the reporting and review requirements set out in Schedule I.",
                        "spaces": 12,
                        "numbering": "2.12.2"
                    },
                    {
                        "content": "be at all times courteous and business like in its contact with the Customers;",
                        "spaces": 12,
                        "numbering": "2.12.3"
                    },
                    {
                        "content": "use its reasonable commercial endeavours to comply with any reasonable and lawful directions, orders and instructions which {P1_Name} may from time to time give to it in accordance with or to give effect to the provisions of this Agreement;",
                        "spaces": 12,
                        "numbering": "2.12.4"
                    },
                    {
                        "content": "identify, procure and keep in force all permits, certificates, licences, approvals, authorisations and consents which may be necessary in connection with the performance of its obligations under this Agreement;",
                        "spaces": 12,
                        "numbering": "2.12.5"
                    },
                    {
                        "content": "in performing its obligations under this Agreement, ensure that it is knowledgeable about and shall continue to be knowledgable about all Regulatory Requirements and that it shall comply with all Regulatory Requirements and (i) maintain evidence of its compliance with Regulatory Requirements, (ii) take all necessary steps required to comply with such Regulatory Requirements promptly upon becoming aware it is not so complying,  and (iii) take all necessary steps to remedy any previous breaches of such Regulatory Requirements;",
                        "spaces": 12,
                        "numbering": "2.12.6"
                    },
                    {
                        "content": "where permitted to do so, promptly notify {P1_Name} in the event that a regulatory body who regulates {P1_Name} or the {P2_Name} conducts an audit or investigation of the {P2_Name} and disclose to {P1_Name} (subject always to the provisions of confidentiality set out at Clause 7) details of any adverse regulatory findings; and",
                        "spaces": 12,
                        "numbering": "2.12.7"
                    },
                    {
                        "content": "co-operate with {P1_Name} and assist them in their dealings with Regulatory Authorities to the extent reasonably required in relation to this Agreement including implementing such measures as are reasonably necessary and appropriate to effect compliance with Regulatory Requirements.",
                        "spaces": 12,
                        "numbering": "2.12.8"
                    }
                ]
            },
            {
                "content": "{P1_Name} acknowledges and accepts that the {P2_Name} may give advice and assistance and provide services and products beyond the scope of the Debt Management Plan to Customers and that the {P2_Name} will not disclose any Customer Data to {P1_Name} without the Customer’s prior consent (which the {P2_Name} is under no obligation to seek).",
                "spaces": 8,
                "numbering": "2.13"
            },
            {
                "content": "Any failure or inability of a Customer to agree to or comply with a Debt Management Plan or any other advice or assistance given by the {P2_Name} pursuant to this Agreement shall not cause the {P2_Name} to be in breach of the terms of this Agreement and shall not prevent the {P2_Name} from providing advice for debt negotiations, counselling and management solutions outside of the Services.",
                "spaces": 8,
                "numbering": "2.14"
            },
            {
                "content": "The Parties acknowledge that the {P2_Name} is not acting as an agent of {P1_Name} and that it is not a debt collection agent of {P1_Name}.",
                "spaces": 8,
                "numbering": "2.15"
            }
        ]
    },
    {
        "content": "Rights and Obligations of {P1_Name}",
        "spaces": 4,
        "numbering": "3.",
        "children": [
            {
                "content": "During the term of this Agreement {P1_Name} shall provide such information and assistance as is reasonably required for the {P2_Name} to perform its obligations under this Agreement.",
                "spaces": 8,
                "numbering": "2.16"
            },
            {
                "content": "For the period of six months from termination or expiry of this Agreement {P1_Name} shall not, without the prior written agreement of the {P2_Name}, employ or engage on any basis or offer such employment or engagement to any of the {P2_Name}’s personnel provided that employment or engagement of any member of the {P2_Name}’s personnel pursuant to a bona fide recruitment campaign shall not be a breach of this clause.",
                "spaces": 8,
                "numbering": "2.17"
            },
            {
                "content": "{P1_Name} represents and warrants that:",
                "spaces": 8,
                "numbering": "2.18",
                "children": [
                    {
                        "content": "it has the requisite power and authority required by any applicable law or otherwise to enter into this Agreement and to carry out the obligations contemplated by the Agreement reliably and professionally and that the execution and performance of this Agreement has been duly authorised by the required corporate action by {P1_Name};",
                        "spaces": 12,
                        "numbering": "2.18.1"
                    },
                    {
                        "content": "it has and shall maintain during the continuance of this Agreement all necessary rights, licences and consents necessary to provide the Customer Information to the {P2_Name} and to perform its obligations under this Agreement.",
                        "spaces": 12,
                        "numbering": "2.18.2"
                    }
                ]
            },
            {
                "content": "If {P1_Name} notifies the {P2_Name} in writing that amendments are required to be made to this Agreement (including any Schedule hereto) to ensure {P1_Name}’s compliance with its obligations to a Regulatory Authority and/or any Regulatory Requirements (including changes required in order to comply with any rules or guidance (including guidance as to interpretation of such rules) issued or published by or on behalf of such Regulatory Authorities or coming into force from time to time), the {P2_Name} shall be obliged to make such amendments as soon as reasonably practicable and in shall use reasonable commercial endeavours to ensure that such changes are made in sufficient time so as to ensure that {P1_Name} is complying with such obligations.",
                "spaces": 8,
                "numbering": "2.19"
            },
            {
                "content": "In the event the {P2_Name} is unable to comply with any amendments as notified to it by {P1_Name} pursuant to Clause 3.1 or fails to comply within a reasonable time then {P1_Name} may terminate this Agreement immediately.",
                "spaces": 8,
                "numbering": "2.20"
            },
            {
                "content": "{P1_Name} shall comply with the reporting and review requirements set out in Schedule I.",
                "spaces": 8,
                "numbering": "2.21"
            },
            {
                "content": "Notwithstanding Clause 2.21 above {P1_Name} shall not make any changes to a Customer’s {P1_Name} account without direct contact with the Customer.  For the avoidance of doubt the Permission shall only relate to the provision of information regarding a Customer’s {P1_Name} account.",
                "spaces": 8,
                "numbering": "2.22"
            }
        ]
    },
    {
        "content": "Conditions",
        "spaces": 4,
        "numbering": "4.",
        "children": [
            {
                "content": "It is a condition of this Agreement that each party is entitled to enter into this Agreement and to perform its obligations set out herein.",
                "spaces": 8,
                "numbering": "2.23"
            }
        ]
    },