YARN - Yet Another Resource Negotiator

YARN =  Yet Another Resource Negotiator

Name node is  having what all are the total blocks, what all are data nodes etc..   at the same time name node allocating resources.  Name node need to decide which resource for which user etc.


YARN is the Hadoop  processing layer that contains.

• A resource manager
• A job scheduler

YARN  allows multiple data processing engines to run on  a single Hadoop cluster.

• Batch Programs (eg., spark,  MapReduce)
• Interactive SQL (e.g. Impala)
• Advanced Analytics ( e.g. Spark, Impala)
• Streaming ( e.g. Spark Streaming)

YARN  Daemons

Resource Manager ( RM)

• Runs on master Node
• Global resource scheduler
• Arbitrates system resources between competing applications.
• Has a pluggable scheduler to support different algorithms ( capacity, fair scheduler, etc.)

Node Manager ( NM)  -  It's logical entity.

• Runs on slave nodes.
• Communicates with RM

Running an Application in YARN

Containers

• Created by the RM upon request.
• Allocate a certain amount of resources(memory, CPU) on a slave node.
• Applications run in one or more containers.

Application Manager ( AM)

• One per application
• Framework/application specific.
• Runs in  a  container.
• Requests more containers to run application tasks.

Cluster Metrics :

How many apps submitted

How many aps pending

How many apps running

How many apps completed

How many containers running

How much memory used ?

How much memory  total ?

How munch memory reserved ?

What are the active nodes ?

Spring DAO

If already JDBC is there why we  should use  Spring DAO ?

JDBC Steps :
------------------

1.  Load the Driver class.
2.  Get the Connection Object
3.  Create the statement/prepared statement/callable statement.
4.  Prepare the SQL Query
5.  Handle exceptions.
6.  Handling Resultset
7.  Close the connection


Note :  For every Java application step 1, 2, 3, 5, 7  are common,  boiler plate code will be taking care by frame work, only has to take care of SQL Query and Handling result set.


JDBC Template  :

1.  DriverClass, Uname, pwd

Instead Developer will write  boiler plate code and closeconnection manually approach then Spring provides module DAO.
In DAO for jdbc there is a template is "JDBCTemplate".  It will take care steps 1-3 and it will also handle resources like manage the connection objects.

Also  JDBCTemplate configurations can be   take care of  in two ways:

a)  XML Approach



<beans>
    <bean id = "org.spf.dao.JDBTemplate">
                ------
                -----

      </bean>
</beans>

     

b)  Java Configuration                                                

@configuration

public class javaconfig{

@Bean

public JDBCTemplate getTemplate(){

JDBCTemplate  template = new JDBCTemplate();

return jdbcTemplate:

}

Also using JDBCTemplate  -  we can  use  SQL and PLSQL to pass SQL queries.

a)  SQL  -  Using this approach   we ca call any one of the method and pass the sql query and get he response.

queryForObject()
queryForString()
queryForInt()
queryForxxx()
....

b) PLSQL  -

a)    IN and OUT  Params  then our DAO class should extends StoredProcedure  using -   execute()
b)   IN and OUT params + resultset hen our DAO class should extends StoredProcedure and implements RowMapper

         --execute()   -  execute  the SP and provides OUT param response
         --mapRow()  -  Handling Resultset.
       

Spring DAO steps to invoke the SP :

Couch base

Syllabus 

What is CouchBase

What is CouchBase

COUCHBASE is a Document Oriented "NOSQL" Database having below features.

• Open Source
• Distributed
• Fault Tolerant
• Persistent  -  Storing data in memory and disc.
• Replicated

• Schema Free & Flexible(Document Size)
• High Performance
• Easy to scale.

Components :    4 nodes running Couch Base services

VBuckets are  nothing but  "PARTITIONS "

Example  VBucket

---------------------------------------------------------------------------------------------------------------

FLOW  ->

JSON  Document  ->

                                  UNIQUE  KEY --->

                                                                    VBucket  ---------->

                                                                                                   Bucket --->  Node  ----> CLUSTER.

---------------------------------------------------------------------------------------------------------------

Editions  Availability

->  Community (Free) and Enterprise Editions are available.

-->  Enterprise edition free for 2 nodes, support license required for large cluster.

-->  Administred via WEB UI, CLI or REST Ful API.

-->  Available for Windows, Linux and Mac.

• Document oriented "NOSQL" Database having features
• Pre-requisites

• System Requirement
• Download

• 
  
• User Requirement

• $hostname
• $userdel -r couchbase
• $userdel -r couchbase
• $useradd couchbase
• passwd couchbase
• $id couchbase
• vi /etc/sudoers

cp .bashrc /home/couchbase

sudo su - couchbase

• Got to  http://www.couchbase.com/nosql-databases/downloads
• Select the Package version and your platform (Linux, Windows, Mac).
• It will download an 80 MB rpm package file - 'couchbase-server-enterprise-4.0.0-centos6.x86_64.rpm'
• Copy the rpm file to your Linux machine.
• You can directly download the package using wget -
•  wget  http://packages.couchbase.com/releases/4.0.0/couchbase-server-enterprise-4.0.0-centos6.x86_64.rpm

Documentation for reference :

• Administrators  - https://docs.couchbase.com/server/5.5/install/get-started.html
• Sample code or projects - https://github.com/couchbaselabs/mobile-training-todo/tree/feature/2.0

• 
  
• 
  
• Download
• Single Node Implementation

• Multi Node Implementation

• Using WEB UI
• Using Couchbase-CLI

• Walkthrough of the COUCHBASE WEB UI.
• Important Administration Commands.

VBucket

The overall key space is partitioned into 1024 logical storage unit called "Virtual buckets" ( or vBucket).
vBucket are distributed across machines within the cluster via a map that is shared among servers in the cluster as well as  the client library.
The basic unit of dat storage in Couchbase is

JSON Document   ->  Unique Key  --->Vbucket  ->Bucket  -->  Node --->Cluster

Manager - Responsibilities

Project Manager

Qualification:

Science / Engineering Graduate

Responsibility:

Requirements Gathering:

• Participate in requirement gathering JAD sessions and support documentation where applicable (business, functional, UI, usability, data, compliance, data migration, NFR, 3rd party tools & products).
• Raise clarifications with customer / onsite to eliminate any ambiguity on the scope of work assigned from the Client / Project Cocoordinator.
• track to closure any open clarifications with customer / onsite.
• understand and translate requirements into design.
• (FOR ONSITE): conduct meetings with customer SMEs, BAs and architect to understand and clarify functional and technical requirements.

Resource/ Efforts Estimation and Monitoring:

• Provide input on technical effort estimation to arrive at resourcing plans.• participate in management reviews and teleconferences and report weekly status and walkthrough to the Project Manager / Client.• Followup with customer POC to resolve team issues and dependencies.• raise concerns regarding changes in timelines and expectations with customer point of contact to resolve the same.

Design & Analysis:

• Develop proof of concept as designed by architect.• setup development environment for team to work in.• break up the modules considering technical aspects.• Understand interdependencies of the various components in work packages that are being developed.• Create design documents with the support of the architects for the assigned scope.• Work with architect to understand the architecture of the application, high level approaches and ensure that the design is in line with overall direction.• conduct peer review, as required.• Organize design review sessions with various stakeholders to validate design is as per requirements.

Coding:

• Identify the standards applicable for the work packages / components.
• Familiarize the team with development environment and CTS and customer spectific coding coding standards, guidelines and best practices to be followed through sessions and document sharing.
• Review the code created by team and provide review comments to the developers.
• develop code for core / critical module(s) needed for the project
.• Contribute towards development of framework.
• Develop/ review tier coding, business layer coding, interface development, service development, creation of stored procedures etc as applicable for the work package / project
.• Write efficient queries and understand logical and physical data models.
• follow up with Developers to ensure integration of components (input / output)
.• troubleshoot critical technical issues.
• Handle escalations related to critical technical issues.
• create detailed tasks for the scope and share timelines for the same with the team.
• track the status of the tasks assigned, maintain detailed task information for the work packages owned and do periodic reporting.
• conduct technical review and audit.
• Create quality check points for all deliverables.
• estimate at task level in the area of work concerned, measure and report the actual effort against the estimated effort and ensure task progress reporting in C20 or any other applicable tool.
• Prepare and report status for the scope responsible.
• conduct peer reviews in coding and testing phase.

Testing:

• Develop comprehensive unit testing cases for modules developed (as required) and unit test plans.
• execute unit testing.
• Create unit test logs.
• prepare unit test strategy, test plan document and data requirements.
• automate unit testing and contributing to automation of unit testing.
• Perform developer integration testing by integrating the components developed by each individual team member and ensure that the integration for the teams work has been completed before the next phase of testing (system testing).
• Signoff on test cases created by team.• discuss results / actions from NFR testing and signoff on the same for system testing / NFR.

Defect Management:

• Participate in defect triage meetings with the team and confirm the defects identified, highlight any new change request to the agreed scope and identify actions to reduce defects.
• Monitor rework carried out on the code based on review comments from code review / defects raised in unit testing, peer testing, SIT, UAT testing or any other relevant testing phase for core / critical modules
.• Follow up with the team to close out on the defects.
• identify ways to prevent defects.• Participate in transition of knowledge and to support organization with any defects raised post production as required.

Process Improvements and Adherence:

• Create training plan for team as identified in the training plan for the project.• Plan and participate in innovation sessions
.• identify and implement innovative processes to reduce manual intervention and improve productivity based on technology and domain expertise.
• Prepare business case to quantify process improvement (efforts saved etc).• Identify/ evaluate opportunities for valueadds and implement valueadds for the project
.• Participate in front end delivery audit(s).
• highlight and identify technical risks to project manager.

Knowledge Management:

• Contribute towards updating knowledge assets, user manual, online help document, installation scripts and installation manual and leverage the same.
• Support process audit activities.
• Conduct knowledge transfer sessions for new joiners in the project team
.• conduct training through academy (if approved)
.• Guide and mentor team junior members from a technical perspective to ensure quality delivery of the components.

People management:

• Engage with team.
• resolve conflict within team.
• Define the teams goals.
• set performance expectations.
• conduct appraisals.
• set stretch targets and track performance against the same periodically.
• Identify, groom, reward and develop high performers.
• create training plans for team.
• Conduct technical evaluation for building team and share inputs with supervisor / project manager.

Functional Skills

SNo	Functional Competency	Functional Skills	Proficiency Level *
1	Technology Concepts		PL3
2	Solution Structuring	Pursuit Management	PL2
3	Solution Structuring	Solution Analysis	PL2
4	Solution Structuring	Financial management	PL2
5	Design Pattern-Tools & Princ.	ETL Concepts and Principles	PL3
6	Design Pattern-Tools & Princ.	BI and OLAP concepts & Princ.	PL3
7	Design Pattern-Tools & Princ.	Datawarehousing Concepts	PL3
8	Design Pattern-Tools & Princ.	Software Design Concepts	PL3
9	Software Development Processes	Product Lifecycle Management	PL3
10	Software Development Processes	SDLC Details	PL3
11	Software Development Processes	SDLC Overview	PL3
12	Estimation & Resource Planning	PBI estimation	PL2
13	Estimation & Resource Planning	Estimation in Agile	PL2
14	Estimation & Resource Planning	Activity Sequencing	PL2
15	Estimation & Resource Planning	Activity Definition	PL2
16	Estimation & Resource Planning	Resourcing	PL2
17	Estimation & Resource Planning	Scheduling	PL2
18	Estimation & Resource Planning	Estimation	PL2
19	Estimation & Resource Planning	Resource Planning	PL2
20	Domain-Industry Knowledge		PL2
21	Test & Defect Management	Test Defect Management Metric	PL2
22	Test & Defect Management	Defect Reports	PL2
23	Test & Defect Management	Defect Analysis	PL2
24	Test & Defect Management	High Maturity Processes	PL2
25	Test & Defect Management	Test Execution	PL2
26	Test & Defect Management	Test Design	PL2
27	Test & Defect Management	Testing Process	PL2
28	Test & Defect Management	Test Strategizing	PL2
29	Test & Defect Management	Defect Life Cycle Process	PL2
30	Test & Defect Management	Test planning	PL2
31	Test & Defect Management	Defect Prevention	PL2
32	Release Mgmt-ProcessTool&Tech	CM Tools & Techniques	PL3
33	Configuration Management	CM Tools & Techniques	PL3
34	Knowledge Management	Planning Knowledge Management	PL2
35	Knowledge Management	Knowledge Mgmt Process & Tools	PL2
36	Requirements Management	Strategy Analysis	PL3
37	Requirements Management	Requirement Management Tools	PL3
38	Requirements Management	Reqm-Business Case Analysis	PL3
39	Requirements Management	Requirements Understanding	PL3
40	Requirements Management	Requirement Analysis	PL3
41	Requirements Management	Scope Verification	PL3
42	Requirements Management	Scope Definition	PL3
43	Requirements Management	Scope Control	PL3
44	Requirements Management	Requirements Verification	PL3
45	Requirements Management	Requirements Modeling	PL3
46	Requirements Management	Change Readiness Assessment	PL3
47	Requirements Management	Capability Analysis	PL3
48	Requirements Management	Requirements Elicitation	PL3
49	Project Management	T&M to MS Framework	PL2
50	Project Management	Risk Identification & Trckg	PL2
51	Project Management	Program Contract Management	PL2
52	Project Management	Business Case Preparation	PL2
53	Project Management	PM Tools	PL2
54	Project Management	Initiation	PL2
55	Project Management	Account & Project Management	PL2
56	Project Management	Planing-Business Case Analysis	PL2
57	Project Management	Planning Knowledge Management	PL2
58	Project Management	Test Defect Management Metric	PL2
59	Project Management	Agile Tools	PL2
60	Project Management	Solution Analysis	PL2
61	Project Management	Operations & Process Mgmt	PL2
62	Project Management	High Maturity Processes	PL2
63	Project Management	Knowledge Mgmt Process & Tools	PL2
64	Project Management	CM Tools & Techniques	PL2
65	Project Management	Value-adds & Best practices	PL2
66	Project Management	Pricing types	PL2
67	Project Management	Planning Process	PL2
68	Project Management	Financial Planning	PL2
69	Project Management	Stakeholder communication	PL2
70	Project Management	Stakeholder Identification	PL2
71	Project Management	Scope Definition	PL2
72	Project Management	Risk Response	PL2
73	Project Management	Risk Reassessment	PL2
74	Project Management	Risk Planning	PL2
75	Project Management	Quantitative Risk Analysis	PL2
76	Project Management	Qualitative Risk Analysis	PL2
77	Project Management	Development methodologies	PL2
78	Project Management	Delivery Excellence	PL2
79	Project Management	Cost Control	PL2
80	Project Management	Activity Sequencing	PL2
81	Project Management	Activity Definition	PL2
82	Project Management	Project Closure	PL2
83	Project Management	Application Lifecycle Mgmt	PL2
84	Project Management	Resourcing	PL2
85	Project Management	Lean Six Sigma	PL2
86	Project Management	Scheduling	PL2
87	Project Management	Financial management	PL2
88	Project Management	Account Management	PL2
89	Project Management	Project Execution & Control	PL2
90	Project Management	Estimation	PL2
91	Project Management	Quality Management	PL2
92	Project Management	Resource Planning	PL2
93	Project Management	Procurement Management	PL2

Behavioral Skills

SNo	Primary Skill	Proficiency Level *
1	Build Collaborative Relations	PL2
2	Managing Teams	PL2
3	Execution Excellence	PL2
4	Managing Change	PL2
5	Client Focus	PL2
6	Strategic Orientation	PL2
7	Effective Communication	PL2
8	Innovation	PL2

* Proficiency Legends

Proficiency Level	Generic Reference
PL1	The associate has basic awareness and comprehension of the skill and is in the process of acquiring this skill through various channels.
PL2	The associate possesses working knowledge of the skill, and can actively and independently apply this skill in engagements and projects.
PL3	The associate has comprehensive, in-depth and specialized knowledge of the skill. She / he has extensively demonstrated successful application of the skill in engagements or projects.
PL4	The associate can function as a subject matter expert for this skill. The associate is capable of analyzing, evaluating and synthesizing solutions using the skill.