Merge pull request #224 from razorpay/release/1.5.13

CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to this project will be documented in this file. The format
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [UNRELEASED][unreleased]
+## [1.5.13][1.5.13]
+- [upi] Vijaya Bank and Dena Gujarat Gramin Bank are no more
+- [upi] 7 new banks now support UPI
+- Metadata update for new release
+- Sanitizes most text fields to remove special characters that show up from encoding errors. Fixes #29, #32
+- Start parsing contact numbers from NEFT sheet as well. Published in E.164 wherever possible
+- Changes some empty fields to null instead of "NA"
+
+## [1.5.12][1.5.12]
+
+- Only metadata changes in this release.
+- Data corrections to account for broken alignment in RBI's RTGS spreadsheet
+- Improved support for Contact details that are sourced from RTGS dataset. CONTACT details are returned in E.164 format
 
 ## [1.5.11][1.5.11]
 ## Changed
@@ -183,7 +196,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 -   Removes some data formats (YAML/Large JSON) for cleaner code. If you were using them, please let create an issue.
 
-[unreleased]: https://github.com/razorpay/ifsc/compare/1.5.11...HEAD
+[unreleased]: https://github.com/razorpay/ifsc/compare/1.5.13...HEAD
+[1.5.13]: https://github.com/razorpay/ifsc/releases/tag/1.5.13
+[1.5.12]: https://github.com/razorpay/ifsc/releases/tag/1.5.12
 [1.5.11]: https://github.com/razorpay/ifsc/releases/tag/1.5.11
 [1.5.10]: https://github.com/razorpay/ifsc/releases/tag/1.5.10
 [1.5.9]: https://github.com/razorpay/ifsc/releases/tag/1.5.9

README.md

@@ -4,9 +4,9 @@ This is part of the IFSC toolset released by Razorpay.
 You can find more details about the entire release at
 [ifsc.razorpay.com](https://ifsc.razorpay.com).
 
-[![wercker status](https://app.wercker.com/status/bc9b22047e1b8eb55ce98ba451d7b504/s/master 'wercker status')](https://app.wercker.com/project/byKey/bc9b22047e1b8eb55ce98ba451d7b504) [![](https://images.microbadger.com/badges/image/razorpay/ifsc:1.5.11.svg)](https://microbadger.com/images/razorpay/ifsc:1.5.11) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)
+[![wercker status](https://app.wercker.com/status/bc9b22047e1b8eb55ce98ba451d7b504/s/master 'wercker status')](https://app.wercker.com/project/byKey/bc9b22047e1b8eb55ce98ba451d7b504) [![](https://images.microbadger.com/badges/image/razorpay/ifsc:1.5.13.svg)](https://microbadger.com/images/razorpay/ifsc:1.5.13) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)
 
-[![](https://images.microbadger.com/badges/version/razorpay/ifsc:1.5.11.svg)](https://microbadger.com/images/razorpay/ifsc:1.5.11) [![npm version](https://badge.fury.io/js/ifsc.svg)](https://badge.fury.io/js/ifsc) [![Gem Version](https://badge.fury.io/rb/ifsc.svg)](https://badge.fury.io/rb/ifsc) [![PHP version](https://badge.fury.io/ph/razorpay%2Fifsc.svg)](https://badge.fury.io/ph/razorpay%2Fifsc) [![Hex pm](http://img.shields.io/hexpm/v/ifsc.svg)](https://hex.pm/packages/ifsc)
+[![](https://images.microbadger.com/badges/version/razorpay/ifsc:1.5.13.svg)](https://microbadger.com/images/razorpay/ifsc:1.5.13) [![npm version](https://badge.fury.io/js/ifsc.svg)](https://badge.fury.io/js/ifsc) [![Gem Version](https://badge.fury.io/rb/ifsc.svg)](https://badge.fury.io/rb/ifsc) [![PHP version](https://badge.fury.io/ph/razorpay%2Fifsc.svg)](https://badge.fury.io/ph/razorpay%2Fifsc) [![Hex pm](http://img.shields.io/hexpm/v/ifsc.svg)](https://hex.pm/packages/ifsc)
 
 ## Dataset
 

ifsc.gemspec

@@ -3,8 +3,8 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |s|
   s.name        = 'ifsc'
-  s.version     = '1.5.11'
-  s.date        = '2021-02-17'
+  s.version     = '1.5.13'
+  s.date        = '2021-05-10'
   s.summary     = 'IFSC code database to help you validate IFSC codes'
   s.description = 'A simple gem by @razorpay to help you validate your IFSC codes. IFSC codes are bank codes within India'
   s.authors     = ['Abhay Rana', 'Nihal Gonsalves']

mix.exs

@@ -4,7 +4,7 @@ defmodule IFSC.Mixfile do
   def project do
     [
       app: :ifsc,
-      version: "1.5.11",
+      version: "1.5.13",
       elixir: "~> 1.5",
       start_permanent: Mix.env == :prod,
       description: description(),

package.json

@@ -1,6 +1,6 @@
 {
   "name": "ifsc",
-  "version": "1.5.11",
+  "version": "1.5.13",
   "description": "This is part of the IFSC toolset released by Razorpay. You can find more details about the entire release at [ifsc.razorpay.com](https://ifsc.razorpay.com). Includes only a validation library as of now.",
   "main": "src/node/index.js",
   "directories": {

scraper/Gemfile.lock

@@ -1,7 +1,7 @@
 GEM
   remote: https://rubygems.org/
   specs:
-    mini_portile2 (2.5.0)
+    mini_portile2 (2.5.1)
     nokogiri (1.11.3)
       mini_portile2 (~> 2.5.0)
       racc (~> 1.4)

scraper/scripts/methods.rb

@@ -8,6 +8,7 @@
 require 'nokogiri'
 require 'open-uri'
 require './methods_nach'
+require './utils'
 
 HEADINGS_INSERT = %w[
   BANK
@@ -22,6 +23,8 @@
 
 # These are not all the known states
 # because the usage is quite limited for now
+# TODO: Change this to a accurate mapping
+# And store statecode instead
 KNOWN_STATES = [
   'ANDHRA PRADESH',
   'DELHI',
@@ -74,18 +77,24 @@ def parse_neft(banks)
     headers = CSV.foreach("sheets/NEFT-#{sheet_id}.csv", encoding: 'utf-8', return_headers: false, headers: true, skip_blanks: true) do |row|
       row = row.to_h
       scan_contact = row['CONTACT'].to_s.gsub(/[\s-]/, '').scan(/^(\d+)\D?/).last
-      row['CONTACT'] = scan_contact.nil? || (scan_contact == 0) || (scan_contact == '0') || (scan_contact.is_a?(Array) && (scan_contact == ['0'])) ? nil : scan_contact.first
+      row['CONTACT'] = parse_contact(row['STD CODE'], row['CONTACT'])
 
       row['MICR'] = row['MICR CODE']
       row.delete 'MICR CODE'
       row.delete 'STD CODE'
-      row['ADDRESS'] = row['ADDRESS'].to_s.strip
+
+      row['ADDRESS'] = sanitize(row['ADDRESS'])
+      row['BRANCH'] = sanitize(row['BRANCH'])
+      row['STATE'] = sanitize(row['STATE'])
+      row['DISTRICT'] = sanitize(row['DISTRICT'])
+      row['CITY'] = sanitize(row['CITY'])
+
       row['IFSC'] = row['IFSC'].upcase.gsub(/[^0-9A-Za-z]/, '')
       codes.add row['IFSC']
       row['NEFT'] = true
 
-      # This hopefully is overwritten by RTGS dataset
-      row['CENTRE'] = 'NA'
+      # This hopefully is merged-in from RTGS dataset
+      row['CENTRE'] = nil
       bankcode = row['IFSC'][0..3]
 
       if banks[bankcode] and banks[bankcode].key? :upi and banks[bankcode][:upi]
@@ -104,27 +113,9 @@ def parse_neft(banks)
   data
 end
 
-# Some rows have last 2 columns shifted by 2
-# Check for numeric values of STATE in RTGEB0815.xlsx for examples
-# This checks and fixes those
-def fix_row_alignment_for_rtgs(row)
-  # List of recognized states
-  unless KNOWN_STATES.include? row['CITY2'].to_s.strip
-    log "#{row['IFSC']} has an unknown state (#{row['CITY2']}), please check"
-    exit 1
-  end
-  # Start right shifting from the right-most column
-  row['PHONE'] = row['STD CODE']
-  # Move STATE's numeric value to STD CODE
-  row['STD CODE'] = row['STATE']
-  row['STATE'] = row['CITY2']
-  # Fix CITY2 value by duplicating CITY1
-  row['CITY2'] = row['CITY1']
-  return row
-end
-
 # Parses the contact details on the RTGS Sheet
-def rtgs_parse_contact(std_code, phone)
+# TODO: Add support for parsing NEFT contact data as well
+def parse_contact(std_code, phone)
   scan_contact = phone.to_s.gsub(/[\s-]/, '').scan(/^(\d+)\D?/).last
   scan_std_code = std_code.to_s.gsub(/[\s-]/, '').scan(/^(\d+)\D?/).last
 
@@ -139,10 +130,16 @@ def rtgs_parse_contact(std_code, phone)
   # If we have an STD code, use it correctly
   # Formatting as per E.164 format
   # https://en.wikipedia.org/wiki/E.164
+  # if possible
   if std_code
     return "+91#{std_code}#{contact}"
-  elsif contact
+  # If it looks like a mobile number
+  elsif contact and contact.size > 9
     return "+91#{contact}"
+  # This is a local number but we don't have a STD code
+  # So we return the local number as-is
+  elsif contact
+    return contact
   else
     return nil
   end
@@ -165,7 +162,7 @@ def parse_rtgs(banks)
         row = fix_row_alignment_for_rtgs(row)
       end
 
-      row['CONTACT'] = rtgs_parse_contact(row['STD CODE'], row['PHONE'])
+      row['CONTACT'] = parse_contact(row['STD CODE'], row['PHONE'])
 
       # There is a second header in the middle of the sheet.
       # :facepalm: RBI
@@ -192,14 +189,15 @@ def parse_rtgs(banks)
         log "Second Entry found for #{row['IFSC']}, discarding", :warn
         next
       end
-      row['ADDRESS'] = row['ADDRESS'].to_s.strip
+      row['ADDRESS'] = sanitize(row['ADDRESS'])
+      row['BRANCH'] = sanitize(row['BRANCH'])
       row['RTGS'] = true
       # This isn't accurate sadly, because RBI has both the columns
       # all over the place. As an example, check LAVB0000882 vs LAVB0000883
       # which have the flipped values for CITY1 and CITY2
-      row['CITY'] = row['CITY2']
-      row['CENTRE'] = row['CITY1']
-      row['DISTRICT'] = row['CITY1']
+      row['CITY'] = sanitize(row['CITY2'])
+      row['CENTRE'] = sanitize(row['CITY1'])
+      row['DISTRICT'] = sanitize(row['CITY1'])
       # Delete rows we don't want in output
       # Merged into CONTACRT
       row.delete('STD CODE')

scraper/scripts/releasenotes.php

@@ -16,6 +16,7 @@
 $common = array_intersect($plus, $minus);
 $plus = array_diff($plus, $common);
 $minus = array_diff($minus, $common);
+
 $summary = [];
 
 foreach ($plus as $ifsc){
@@ -26,6 +27,7 @@
     $summary[$bank] +=1;
 }
 foreach ($minus as $ifsc) {
+    $bank = substr($ifsc, 0, 4);
     if(!isset($summary[$bank])) {
         $summary[$bank] = 0;
     }

scraper/scripts/utils.rb

@@ -0,0 +1,30 @@
+def sanitize(str)
+  return nil if str.nil? or str.length==0
+  ["┬ô", "┬û",'┬ö','┬Æ','┬á','┬æ','┬ù','ý','ý','┬á'].each do |pattern|
+    str.gsub!(pattern,' ')
+  end
+  str.gsub!('É','e')
+  str.gsub!('Æ','a')
+  str.gsub!('É','e')
+  # Remove all spaces (including nbsp) at the start and end of the string
+  str.gsub(/\A[[:space:]]+|[[:space:]]+\z/, '')
+end
+
+# Some rows have last 2 columns shifted by 2
+# Check for numeric values of STATE in RTGEB0815.xlsx for examples
+# This checks and fixes those
+def fix_row_alignment_for_rtgs(row)
+  # List of recognized states
+  unless KNOWN_STATES.include? row['CITY2'].to_s.strip
+    log "#{row['IFSC']} has an unknown state (#{row['CITY2']}), please check"
+    exit 1
+  end
+  # Start right shifting from the right-most column
+  row['PHONE'] = row['STD CODE']
+  # Move STATE's numeric value to STD CODE
+  row['STD CODE'] = row['STATE']
+  row['STATE'] = row['CITY2']
+  # Fix CITY2 value by duplicating CITY1
+  row['CITY2'] = row['CITY1']
+  return row
+end

src/banks.json

@@ -6692,7 +6692,8 @@
     "apbs": true,
     "ach_credit": true,
     "ach_debit": true,
-    "nach_debit": false
+    "nach_debit": false,
+    "upi": true
   },
   "LAVB": {
     "code": "LAVB",
@@ -9634,7 +9635,8 @@
     "apbs": false,
     "ach_credit": true,
     "ach_debit": true,
-    "nach_debit": false
+    "nach_debit": false,
+    "upi": true
   },
   "RAEX": {
     "code": "RAEX",
@@ -11793,7 +11795,8 @@
     "apbs": true,
     "ach_credit": true,
     "ach_debit": true,
-    "nach_debit": false
+    "nach_debit": false,
+    "upi": true
   },
   "SURX": {
     "code": "SURX",
@@ -11838,7 +11841,8 @@
     "apbs": false,
     "ach_credit": true,
     "ach_debit": false,
-    "nach_debit": false
+    "nach_debit": false,
+    "upi": true
   },
   "SUTB": {
     "code": "SUTB",
@@ -12187,7 +12191,8 @@
     "apbs": true,
     "ach_credit": true,
     "ach_debit": true,
-    "nach_debit": false
+    "nach_debit": false,
+    "upi": true
   },
   "TCCX": {
     "code": "TCCX",
@@ -13942,8 +13947,7 @@
     "apbs": true,
     "ach_credit": true,
     "ach_debit": true,
-    "nach_debit": false,
-    "upi": true
+    "nach_debit": false
   },
   "VIJX": {
     "code": "VIJX",
@@ -13966,7 +13970,8 @@
     "apbs": true,
     "ach_credit": true,
     "ach_debit": true,
-    "nach_debit": false
+    "nach_debit": false,
+    "upi": true
   },
   "VIMX": {
     "code": "VIMX",

src/patches/banks/upi-enabled-banks.yml

@@ -1,7 +1,8 @@
 ---
-# https://github.com/razorpay/ifsc/issues/126
-# Source: https://www.npci.org.in/upi-live-members
-# Last Updated: Sep 2020 (Count = 168)
+# Source: https://www.npci.org.in/what-we-do/upi/live-members
+# Remember to take a snapshot at https://web.archive.org/
+# when you make changes to this file
+# Last Updated: May 2021 (Count = 221)
 action: patch
 patch:
   :upi: true
@@ -58,7 +59,6 @@ banks:
   - DBSS
   - DCBL
   - DCUX
-  - DEGX
   - DEUT
   - DGBX
   - DLXB
@@ -112,6 +112,7 @@ banks:
   - KPCX
   - KVBL
   - KVGB
+  - LATX
   - LAVB
   - MAHB
   - MAHG
@@ -155,10 +156,12 @@ banks:
   - PUNB
   - PURX
   - PYTM
+  - RACX
   - RATN
   - RJTX
   - RMGB
   - RNSB
+  - RSBL
   - SACB
   - SADX
   - SAGX
@@ -184,7 +187,9 @@ banks:
   - STCB
   - STRX
   - SUBX
+  - SUNB
   - SURY
+  - SUSX
   - SUTB
   - SUVX
   - SVAX
@@ -194,6 +199,7 @@ banks:
   - SYNB
   - TAMX
   - TBSB
+  - TCBX
   - TGBX
   - TJSB
   - TMBL
@@ -211,8 +217,8 @@ banks:
   - VARA
   - VCCX
   - VGBX
-  - VIJB
   - VIJX
+  - VIKX
   - VISX
   - VSBL
   - VSCX