Keystone Initiative for Network Based Education and Research (KINBER) Regional Cyberinfrastructure Plan - 2015

KINBER Cyberinfrastructure Plan
January 26, 2015

KINBER is a not-for-profit corporation established to work with its members to create an environment that fosters collaborative research and promotes the innovative use of digital technologies among educational, healthcare, media and other community anchor institutions to improve the quality of life in communities throughout the Commonwealth. Its mission is to provide broadband connectivity and outstanding services to its current and future members.

Unlike most of its neighboring states, Pennsylvania did not have a statewide broadband network to serve the ever-expanding needs of colleges and universities, healthcare centers, economic development entities and others for whom high speed Internet access is essential.

In 2009, a once-in-a-generation opportunity arose to create such a network with the offering of competitive grants through the National Telecommunications and Information Administration (NTIA). In an historic collaborative effort, the Keystone Initiative for Network Based Education and Research (KINBER), which included representatives from every sector of higher education, healthcare, economic development, and other partners, applied to NTIA for one of the grants to design and manage a 1,600-mile broadband network that would stretch across the Commonwealth. In February 2010, KINBER was awarded a $99.6 million grant by NTIA, which was matched by over $29 million in cash and in-kind contributions from KINBER, contractors and members. The network resulting from this project, the Pennsylvania Research and Education Network (PennREN) was designed in 2010 with construction beginning in 2011, with the network completed on time and within budget in February 2013. PennREN’s mission is to serve the educational and research needs of the K-20 sector in Pennsylvania as well as other nonprofit and research institutions.

In February 2013 the KINBER Board approved a five-year strategic plan outlining the mission, vision, guiding principles and strategic initiatives for the organization and forming the basis for KINBER Cyberinfrastructure Plan.

Current Infrastructure

PennREN was created by lighting a pair of KINBER fibers using ADVA DWDM equipment capable of supporting up to 40 waves over each segment of the network. The backbone can be upgraded to support up to 80 waves through the acquisition of additional hardware. PennREN uses Juniper MX480s to light two 10GE from across the entire network with a third 10GE wave completed by the end of April 2014.

The PennREN backbone consists of four different nodes types, routing, service, access and amplification, that facilitate both the operation of and connecting to the network. The two routing nodes provide general Layer3 routing as well as access to wide area networking services; services nodes are used to connect members to the PennREN backbone; access nodes are fiber paths not on the PennREN backbone that provide easy fiber based access for members; amplification nodes contain DWDM hardware for amplifying PennREN backbone waves and can be turned into access nodes as needed.

Two routing nodes are located in carrier hotels, one in Pittsburgh and one in Philadelphia. These nodes provide gateways to wide area network resources as well as the routing all the PennREN layer3 services, currently PennREN member routes and commodity connectivity. These nodes also support member
connections. Additional services can be configured on the platforms as PennREN grows its service offerings.

Twelve service nodes, also referred to as PennREN Points of Presence (PoPs), on the backbone contain active electronics used to connect members. These nodes support both 1 GE and 10GE direct connections as well as connections through KINBER provisioned dark fiber.

Access nodes are KINBER fiber locations that are not yet PoPs on the PennREN backbone. These locations are typically on the campus of a KINBER member and are lit once the organization decides to utilize PennREN services.

Amplification Nodes are backbone PoPs that only have ADVA DWDM hardware for amplifying the PennREN DWDM waves. These nodes are typically on the campus of a KINBER member and are also used as needed by members to connect to a service node.

Most members connect via 1G or 10G Ethernet to a service node or
an access node switch. Members configure their port to use specific vlan numbers for each service even if they only plan to use one service, which allows the provision of multiple discrete services on a single port. Example services include commodity Internet service, KINBER member exchange, and private layer2 VLANs.

Current Service Offerings

The initial services offerings on PennREN focus on connecting KINBER members to the backbone network and providing a basic set of wave, switched or routed network services, similar to what is offered on other regional and national Research and Education networks. These services include: 1 GE or 10GE port connection, Ethernet point-to-point service, Ethernet point to multipoint, KINBER Member Exchange (KMEX), Commodity Internet Service and member to member services.

The KINBER Member Exchange allows organizations participating on the PennREN network to freely exchange traffic over a layer2 fabric allowing participants to share a common network spread across multiple locations.

Commodity Internet2 services is provided through two redundant 10 GE connections to Cogent connected through the routing nodes in Pittsburgh and Philadelphia. In addition, PennREN connects to other regional networks, including the PAIUNet, the state network for K12 and MAGPI and 3ROX/Drexel, the Internet2 connectors in the region.

Member to member services include access to Internet2 and other wide area R&E networks through the current Pennsylvania Internet2 connectors, 3ROX/Drexel or MAGPI. Other services currently accessible through PennREN connections include high-performance computing and storage at the Pittsburgh Supercomputing Center, video conferencing and educational services through MAGPI and shared scientific resources such as the ATLSS Multi-Directional Experimental Laboratory at Lehigh University.

PerfSONAR

PennREN has deployed perfSONAR platforms at each of the twelve KINBER services nodes. The perfSONAR platforms are part of the PennREN perfSONAR mesh which runs hourly OWAMP and BWCTL tests over a 1 GE interface. In addition, the platforms have 10GE interfaces, used for debugging purposes. The PennREN perfSONAR BWCTL and OWAMP mesh data is publicly available and is used by both KINBER staff and member organization to understand backbone performance.

Future Plans

Over the next five years, KINBER plans to expand the services offered on PennREN to meet the needs of its growing user base. For on-net members, we continue to investigate optics, switches and routers that can cost-effectively connect them to the network. For off-net members, we continue to investigate both fiber based and lit service offerings to connect them to the nearest PennREN access or service node. As we have over the past year, we will continue to partner with our members to do local fiber builds, connecting the local fiber build to the nearest splice point on our network. As a middle mile infrastructure, the initial network build was not able to reach all the small colleges and universities within the state. Documenting and sharing information on cost-effective ways to gain access to the backbone is critical for enabling these organizations to connect to and utilize the PennREN backbone and services. In addition, we are investigating the use of PennREN access nodes to connect individual communities, such as the current Lancaster County initiative, providing a model that can be used by other community throughout Pennsylvania.

We will expand the backbone network as needed, including adding 10 GE or 100 GE waves based on user demand. A number of the larger university members associated with KINBER have funded research projects with growing network needs, such as the ATLSS Multi-Directional Experimental Laboratory at Lehigh University and the computing and storage resources at the Pittsburgh Supercomputing Center. We will need to support emerging educational applications, such as distance learning between the university members and the community colleges. Recent discussion have indicated that the best-effort shared KMEX service may not meet the requirement of such applications, thus we will have to work with the end-user to design, develop and provision the appropriate type of service. With the implementation science DMZ at CC*NIE funded schools such as Lehigh University and The Pennsylvania State University (PSU) we expect an increase in demand for 10GE connectivity as well as 100GE. As science DMZ implementations, including the use of Data Transfer Nodes, become more common, we expect to see request for greater bandwidth come from all sectors of our community.

We plan to expand the wide area network services currently offered on the PennREN backbone, with a number of new offerings deployed within the next year. While we have peering with some of the regional networks in the area, we will expand this program to peering at 10GE and with neighboring networks such NYSERnet, OARNet and NJEdge. This additional connectivity will enhance research collaborations throughout the region, particular to the smaller colleges and universities who currently do not have easy access to peering infrastructure.

Implementation of IPv6 and InCommon

PennREN has the infrastructure to fully support IPv6 protocols and is working to educate its users about the deployment and transition from IPv4 to IPv6. Many of the institutions connected to PennREN have turned up an IPv6 BGP session. Others have discovered technical problems during the transition process and KINBER engineers have been able to help them work through the issues. Each institution is unique in its needs, configuration and level of expertise.

Pennsylvania has 29 InCommon members representing over 50 campuses throughout the Commonwealth with 46 of those campuses are directly connected to the PennREN network. Lead Engineer Mike Carey is participating in the InCommon Regional Pilot Program, a new support and business model where the regional network is the InCommon member and provides a variety of functions to its downstream members. KINBER plans to leverage the existing expertise within the state, including Lafayette College, PSU and Indiana University of Pennsylvania, to provide input in to the process. As part of the program, we are developing an InCommon Pilot Program Plan for Pennsylvania that we will share with both our university collaborators as well as other regional network participating in the pilot program. KINBER continues to provide educational opportunities related to federated identity and is seen as a leader in facilitating the use of InCommon in Pennsylvania.

Network Operations

The KINBER engineering staff provides the overall design, implementation and trouble shooting support for the PennREN network. This engineering support is augmented through a contract with the GRNOC, based at Indiana University, for Network Operations and Engineering services. The GRNOC provides 24/7 on-call and helpdesk services, support for turn up of new connections and circuits on the PennREN backbone as well as network debugging and diagnostic support.